Troubleshooting Jobs

Connection Issues

connection refused

waiting for connection

If transfers succeed but an error appears in the logs, the storage client may be starting a transfer before the SDCX Server is able to accept connections. In this case, connection refused errors should only occur for a few seconds until the SDCX Server becomes available and the transfer begins.

If transfers fail, these errors can indicate that:

• An endpoint requires a custom transfer IP address

  Verify that the transfer IP addresses are able to connect to each other over the network.

• The SDCX Server is not running on the destination endpoint

  After logging into the server via a terminal, run the status command to verify that the Jet Server is running: systemctl status jet-server

• A firewall is blocking the required ports

  To isolate the issue:

  1. Run the netstat command to check that the required ports are open and in a LISTEN state:

  netstat -tnl 2. Telnet to the port over the local loopback adapter to ensure that incoming TCP connections on ports 443, 49221 and 50221 are being accepted:
  telnet <host> <port_number> 3. Use the dds_cnctst -udp command to test access to UDP port 49221. 4. If a waiting for connection error remains unresolved, check that ports using bi-directional connections have sockets that reach an ESTABLISHED state:
  netstat -tan | grep <port_number> If an outbound firewall rule is restricting one or more of the required ports, any inbound connection attempts will be partially accepted by the server but left in a SYN_SENT or SYN_RECV state because the firewall is not allowing the bi-directional TCP handshake to complete.

File Issues

stale NFS file handle

This error message can appear if a file changes during the 10-second waiting period between the beginning of a transfer and when files start moving to the destination.

Once modifications to the file stop, the transfer will resume.

file does not exist

This error message may appear if the file on the source endpoint has been deleted, renamed or moved.

Check the source directory to ensure that the file still exists.

direct stat returned error

This error message indicates that Jet is unable to identify the status of files on the source or destination endpoint. This issue may occur if the endpoints on a route cannot connect, if storage is not connected to an endpoint, or if network storage permissions do not allow access to file status information.

To isolate the issue:

1. Run a connection test on the route. Verify that the endpoints connect successfully over TCP and UDP.
2. Create a job using a single endpoint and transfer content between directories on the endpoint.
3. Verify that network storage permissions allow transfers.

Server Issues

There may be an issue where a server does not appear in the Jet management console when configuring a Jet endpoint after a successful SDCX Server installation.

To isolate the issue, check that the deviceID and keys appear in these files:

/usr/Signiant_Media_Shuttle/bin/tomcat/webapps/ROOT/WEB-INF/config.properties 
/usr/local/etc/signiant/signiant-communication-service-config.json5  
/usr/local/etc/signiant/signiant-communication-service-keys.json5  

If no deviceID is associated with the configuration or service key files, run the dds_admin utility in the bin directory to check connectivity. Verify that the time offset is not more than a few seconds off.

Performance Issues

In some cases, Jet transfer speed may be limited by the number of worker threads allocated. By default, there are 8 worker threads available to pass chunks of data at a time. Increasing the number of worker threads to 16 and the part length to 16Mi on the source and destination endpoints may improve transfer speed.

To change the number of worker threads and the part length:

1. Open the Jet Server configuration file.
   Linux: /usr/local/etc/signiant/jet-server/config.json
   Windows: C:\Program Files\Signiant\Jet Server\config.json
2. Update the storageClientOverrides section to increase worker threads to 16 and part length to 16Mi for the source and the target.

"storageClientOverrides": {
        "enabled": true,
        "flightGateway": "",
        "source": {
            "partLength": "16Mi",
            "partRetryLimit": 10,
            "workerThreads": 16,
            "maxPartLength": "128Mi"
        },
        "target": {
            "partLength": "16Mi",
            "partRetryLimit": 10,
            "workerThreads": 16,
            "maxPartLength": "128Mi"
        }
    }

3. Save the configuration file and restart Jet services:

   Linux: In your terminal, run >./siginit restart.
   Windows: In the Services control panel, right-click on the services and choose Restart.

4. Transfer some new files to determine if the changes have increased the transfer speed.

5. If there is no gain in speed, use iPerf to determine the maximum route transfer speed.