ReFlex.Library

Core Library for Development of applications for Elastic Displays. Created in context of research project ZELASTO (EFRE-project no. 100376687) in cooperation with HTW Dresden.

Table of contents

1. Project setup
2. Project Structure
3. Sensor Modules
4. Specific components
5. Code Examples
6. Configurable Clients (TCP/WebSockets)

Project setup

Frameworks

• .NET 8.0

IDEs

• Microsoft Visual Studio
  • recommended: ReSharper Extension
• JetBrains Rider
• Visual Studio Code

nuget Packages

general

Connectivity

Development

Project Structure

Sensor Modules

implemented Sensor-Hardware

• Intel RealSense R200 (RealSenseR2Module)
• Intel RealSense D435 (RealSenseD435Module)
• Intel RealSense L515 (RealSenseL515Module)
• Microsoft Kinect 2 (Kinect2Module)
• Microsoft Azure Kinect (AzureKinectModule)

Software-Emulator:

• Emulator (EmulatorModule)

Specific components

Filter

Algorithms to process raw depth image for smoothing and mapping.

• cf. Wiki_Filter

BoxFilter

Smoothing depth image based on Box-Filter algorithm.
Parameters:

• Radius: determine the strength of the smoothing

CalibrationFilter

Map depth coordinates (x,y) to screen coordinates based on given calibration matrix. (simnplke transfomation: scale + translate)
Parameters:

• Calibration: 2x2 Matrix to be multiplied with depth value to retrieve corresponding screen coordinate

GaussianFilter

Separated two pass (X, then Y) smoothing algorithm based on gaussian blur.
Parameters:

• Radius: determine the strength of the smoothing

LimitationFilter

removes tracking values outside of desirted viewing frame (e.g. interactive surface)
Parameters:

• LeftBound: left border, values with lower x-index willl be removed
• RighBound: right border, values with higher x-index willl be removed
• UpperBound: top border, values with lower y-index willl be removed
• LowerBound: bottom border, values with higher y-index willl be removed

ThresholdFilter

Filters values which inner z-values are greater than a specified threshold. (== Limitation Filter in z-direction)
Parameters:

• Threshold: max absolute distance from display plane, depth vaues with a distance larger than this value are removed

ValueFilter

removes depth values with specific value from depth image (e.g. remove all zeros) Parameters:

• X,Y,Z: values for filtering: depth value of [x,y,z] is removed

Code Examples

TCP-Client

Basic code example to create a tcp-based client and listen to events from server (cf. ReFlex.Utilities.ConsoleClientDemo/Client.cs)

public class Client
{ 
    // create new TCP-Client
    private readonly IClient _client = new TcpClient();

    public Client()
    {
        // attach to DataReceivedEvent and start listening to address 
        _client.NewDataReceived += HandleDataReceived;
        Run("localhost", 8080);
    }

    // handle events and parse messages from tracking server
    private void HandleDataReceived(object sender, Message message)
    {
        if (message?.MessageString == null)
            return;

        // retrieve list of interactions
        var interactions = SerializationUtils.DeserializeFromJson<List<Interaction>>(message.MessageString);
        
        // iterate over interactions
        interactions.ForEach(interaction =>
        {
            // get time of interaction
            var timeString = DateTime.FromFileTimeUtc(interaction.Time).ToShortTimeString();

            // ... do something        
        });
    }

    public void Run(string address, int port)
    {
        // connect if client is properly initialized
        _client?.Connect(address, port);  
    }

    public void Stop()
    {
        _client?.Disconnect();
    }
}

Configurable Clients (TCP/WebSockets)

• cf. ReFlex.Utilities.ConsoleClientDemo/ ReFlex.Utilities.ConsoleServerDemo
• both programs can be configured using the included app.config:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="NetworkInterface" value="Http"/>
    <add key="Address" value="127.0.0.1"/>
    <add key="Port" value="40000"/>
    <add key="Endpoint" value="/ReFlex"/>

    <!-- values for TCP -->
    <!--<add key="NetworkInterface" value="tcp"/>
    <add key="Address" value="127.0.0.1"/>
    <add key="Port" value="8080"/>
    <add key="Endpoint" value=""/>-->
  </appSettings>
</configuration>

• by default (using the values above), Websocket connection is establishjed at address ws://127.0.0.1/ReFlex, TCP connection via 127.0.0.1:8080;
• Testing websocket connections can easily be done by using the included HTML template in src\Utilities\WebSocketTestHtml (launch.json for Chrome debugging using VS code is included).