Only the video are delayed[kinesis video stream WebRTC]

The delay you are experiencing is not entirely unexpected when using both audio and video streams together. There are a few factors that contribute to the increased latency when transmitting both audio and video simultaneously:

Synchronization: When transmitting both audio and video, the system needs to synchronize the two streams to ensure that the audio and video are properly aligned. This synchronization process can introduce additional latency, especially if there are differences in the processing time or buffering requirements for the two streams.

Buffering and Jitter Management: To maintain a smooth playback experience, both the audio and video streams typically require buffering and jitter management mechanisms. This involves introducing some delay to compensate for network fluctuations and variations in packet arrival times. The buffering and jitter management requirements can be different for audio and video, leading to different latencies for each stream.

Video Encoding and Decoding: Video encoding and decoding processes can be more computationally intensive compared to audio encoding and decoding. This can lead to additional latency in the video pipeline, particularly if the encoding or decoding is not optimized for low-latency scenarios.

Network Conditions: Network conditions, such as bandwidth constraints, packet loss, and network congestion, can also contribute to increased latency, especially for video streams, which typically have higher bandwidth requirements.

The delay of around 4000ms (4 seconds) for the video and 500ms for audio when transmitting both streams simultaneously is not entirely unexpected, but it may be considered high for some real-time communication scenarios.

To reduce the overall latency, you could consider the following strategies:

Optimize Video Encoding Settings: Adjust the video encoding settings, such as the bit rate, key frame interval, and encoding presets, to prioritize low-latency over quality. This can help reduce the encoding and decoding latency for video.

Use Hardware Acceleration: If available, consider using hardware acceleration for video encoding and decoding, as it can significantly improve performance and reduce latency.

Reduce Buffering and Jitter Management: Explore ways to reduce the buffering and jitter management requirements, such as using lower buffer sizes or more aggressive jitter management strategies. However, this may come at the cost of increased potential for stuttering or occasional frame drops.

Network Optimization: Optimize the network conditions by ensuring sufficient bandwidth, minimizing packet loss, and reducing network congestion. This can be achieved through techniques like Quality of Service (QoS) settings, traffic shaping, or dedicated network paths.

Consider Selective Forwarding Approach (SFU): Investigate the possibility of using a Selective Forwarding Unit (SFU) architecture, where the audio and video streams are handled separately and optimized for their respective latency requirements.

There is often a trade-off between latency and other factors, such as video quality, reliability, and resource utilization. The acceptable latency depends on your specific use case and requirements. If the current latency is not acceptable for your real-time communication scenario, you may need to prioritize latency over other factors and make appropriate adjustments to your pipeline configuration and network setup.