Tried to create a video chat using WebRTC, didn't know it would take this long.There weren't many conference using Spring, and it is difficult to apply, it's getting to late, But since we made it, let's get started.
Function definition
Since it is a prototype, the function was made simple
- main
- Users should be able to see all study rooms.
- Create study room
- In order to use video chat, user have to create study room
- create user
- The user must enter the user name to use in the selected study room.
- video chat
- On the basis of information of user and study room, the user have a video chat
Technical definition
- Spring
- webSocket
- Spring is in charge of the signaling server, and full-duplex communication is required for real-time communication between users in the study room. So we used Websocket.
- thymeleaf
- Since it is a prototype, We used it because wanted to procee only with Spring
- webSocket
- MySQL
- It was necessary to store the information of users and rooms.
Implementation
It is a general CRUD except for video chat witch WebRTC applied. Therfore, we would like to explain briefly.
Entity
Entity only has information about study room and the users in thata study room.
StudyRoom.java
@Entity
@AllArgsConstructor
@NoArgsConstructor
@Builder
@Getter
@ToString(callSuper = true)
@Table(name = "studyRooms")
public class StudyRoom {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long room_id;
@Column(nullable = false, columnDefinition = "BINARY(16)")
@Builder.Default
private UUID room_uuid = UUID.randomUUID();
@Column(nullable = false)
private String room_name;
}we added uuid because study room's PK should not be exposed to users.
User.java
@Entity
@ToString(callSuper = true)
@Builder
@Getter
@NoArgsConstructor
@AllArgsConstructor
@Table(name = "users")
public class User {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long user_id;
@Column(nullable = false, columnDefinition = "BINARY(16)")
@Builder.Default
private UUID user_uuid = UUID.randomUUID();
@Column(nullable = false)
private String user_name;
@ManyToOne(fetch = FetchType.LAZY)
@JoinColumn(name = "room_id")
private StudyRoom room;
}
Main
Main page soulds show users what study rooms are available. Therefore, only needs the function of bringing and distributing all study rooms.
MainController.java
@RequiredArgsConstructor
@Controller
public class MainController {
private final StudyRoomService studyRoomService;
@GetMapping("")
public String displayMainPage(Model model) {
model.addAttribute("request", CreateStudyRoomRequest.builder().build());
model.addAttribute("rooms", studyRoomService.findAllStudyRoom());
return "main";
}
}
StudyRoomService.java
@RequiredArgsConstructor
@Service
public class StudyRoomService {
private final StudyRoomRepository studyRoomRepository;
private final UserRepository userRepository;
private final StudyRoomMapper studyRoomMapper;
//...
public List<GetStudyRoomResponse> findAllStudyRoom() {
List<StudyRoom> rooms = studyRoomRepository.findAll();
return studyRoomMapper.toDto(rooms);
}
//...
}
main.html
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:th="http://www.thymeleaf.org">
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Main Page</title>
<!-- Latest minified Bootstrap & JQuery-->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.1.0/css/bootstrap.min.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.1.0/js/bootstrap.min.js"></script>
<!-- Custom styles for this template -->
<link rel="stylesheet" type="text/css" href="/css/main.css"/>
</head>
<body class="text-center">
<!-- Begin page content -->
<main role="main" class="container">
<h1>Simple WebRTC Signaling Server</h1>
<div id="container">
<p>
This part receives a room name (or generates new one), and redirects current user there.
</p>
<div class="row justify-content-md-center">
<div class="input-group col-md-6 mb-3 justify-content-md-center">
<div class="mb-3" th:if="${not #lists.isEmpty(rooms)}" th:attr="data-uid=${room_uuid}">
<label for="rooms-list">Select one of the rooms created:</label><br>
<h4>
<span id="rooms-list" th:each="r : ${rooms}">
<a th:href="@{/{room_uuid}/users(room_uuid=${r.room_uuid})}" th:id="'button-link-'+${r.room_name}">
<button type="button" name="action" th:text="${r.room_name}" th:value="${r.room_uuid}"
class="btn badge badge-primary">
</button>
</a>
</span>
</h4>
</div>
<form th:method="post" th:action="@{/studies}" th:object="${request}" id="form">
<div class="mb-3">
<input class="form-control" type="text" id="room_name" th:field="*{room_name}" placeholder="Enter Study Room Name" required/>
</div>
<div>
<button name="action" value="create" type="submit" class="btn btn-outline-primary">
Create Selected Room</button>
</div>
</form>
</div>
</div>
</div>
</main>
<!--generates unique user identifier stored at 'uuid' input field to pass it to the back-side-->
<script src="/js/main.js"></script>
</body>
</html>
Create study room
If the user enters the study room name in the main page to create study room, the study room is created. Then, by redirecting to the main page, all study rooms, including the created study room, are shown.
StudyRoomController.java
@RequiredArgsConstructor
@RequestMapping("studies")
@Controller
public class StudyRoomController {
private final StudyRoomService studyRoomService;
private final UserService userService;
@PostMapping("")
public String createStudyRoom(@Valid CreateStudyRoomRequest request) {
studyRoomService.createStudyRoom(request);
return "redirect:/";
}
//...
}
StudyRoomService.java
public class StudyRoomService {
//...
public StudyRoom createStudyRoom(CreateStudyRoomRequest request) {
StudyRoom studyRoom = studyRoomMapper.toEntity(request);
return studyRoomRepository.save(studyRoom);
}
//...
}
create user
To enter the study room, the user enters own information(user name). After that, can enter the study room.
UserContorller.java
@RequiredArgsConstructor
@Controller
public class UserController {
private final UserService userService;
@GetMapping("{room_uuid}/users")
public String displayUserInfoPage(Model model, @PathVariable("room_uuid") UUID room_uuid) {
model.addAttribute("request", CreateUserRequest.builder().build());
model.addAttribute("room_uuid", room_uuid);
return "user_info";
}
@PostMapping("{room_uuid}/users")
public String createUser(
@Valid CreateUserRequest request, @PathVariable("room_uuid") UUID room_uuid, Model model) {
User user = userService.createUser(request, room_uuid);
return "redirect:/studies/" + room_uuid + "/" + user.getUser_uuid();
}
}
UserService.java
@RequiredArgsConstructor
@Service
public class UserService {
private final UserRepository userRepository;
private final StudyRoomRepository studyRoomRepository;
private final UserMapper userMapper;
public User createUser(CreateUserRequest request, UUID room_uuid) {
log.info("[UserService][createUser] request : {}", request);
StudyRoom studyRoom =
studyRoomRepository.findStudyRoomByRoom_uuid(room_uuid).orElseThrow(NotFoundStudyRoom::new);
return userRepository.save(userMapper.toEntity(request, studyRoom));
}
//...
}
user_info.html
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:th="http://www.thymeleaf.org">
<html lang="en">
<head>
<meta charset="UTF-8">
<title>UserName</title>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.1.0/css/bootstrap.min.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.1.0/js/bootstrap.min.js"></script>
</head>
<body>
<!-- Begin page content -->
<main role="main" class="container">
<h1>Enter Your Information</h1>
<div id="container">
<form th:method="post" th:action="@{/{room_uuid}/users(room_uuid=${room_uuid})}" th:object="${request}">
<div class="form-group">
<input class="form-control" name="user_name" id="user_name" type="text" placeholder="Enter Your Name"
th:field="*{user_name}" required="required"/>
</div>
<button type="submit" class="btn btn-primary block full-width m-b">Join</button>
</form>
</div>
</main>
</body>
</html>
Video Chat
Join Study Room
Now, let's get started. First of all, enter the selected study room based on the room_uuid and user_uuid.
StudyRoomController.java
@RequiredArgsConstructor
@RequestMapping("studies")
@Controller
public class StudyRoomController {
private final StudyRoomService studyRoomService;
private final UserService userService;
//...
@GetMapping("/{room_uuid}/{user_uuid}")
public String getStudyRoom(
Model model,
@PathVariable("room_uuid") UUID room_uuid,
@PathVariable("user_uuid") UUID user_uuid) {
GetStudyRoomResponse studyRoom = studyRoomService.findStudyRoomByRoom_uuid(room_uuid);
GetUserResponse user = userService.getUser(user_uuid);
model.addAttribute("studyRoom", studyRoom);
model.addAttribute("user", user);
return "chat_room";
}
}
StudyRoomService.java
@RequiredArgsConstructor
@Service
public class StudyRoomService {
private final StudyRoomRepository studyRoomRepository;
private final UserRepository userRepository;
private final StudyRoomMapper studyRoomMapper;
//...
@Transactional
public GetStudyRoomResponse findStudyRoomByRoom_uuid(UUID room_uuid) {
StudyRoom studyRoom =
studyRoomRepository.findStudyRoomByRoom_uuid(room_uuid).orElseThrow(NotFoundStudyRoom::new);
Boolean hasUser = userRepository.checkUsersByRoom_uuid(room_uuid);
return studyRoomMapper.toDto(studyRoom,hasUser);
}
}The response includes hasUser.
GetStudyRoomResponse.java
@Builder
@Getter
public class GetStudyRoomResponse {
private UUID room_uuid;
private String room_name;
private boolean hasUser;
}HasUser checks the presence of other users in the study room. This determines whether or not to connect with other users when entering the sutyd room. If user's alone in the study room, shouldn't try connection.
UserRepository.java
@Repository
public interface UserRepository extends JpaRepository<User, Long> {
@Query("SELECT u FROM User u WHERE u.user_uuid = :user_uuid")
Optional<User> findUserByUserUuid(@Param("user_uuid") UUID user_uuid);
@Query("SELECT u FROM User u JOIN FETCH u.room WHERE u.room.room_uuid =:room_uuid")
List<User> findAllByRoom_uuid(@Param("room_uuid") UUID room_uuid);
@Query("SELECT COUNT(u) > 1 FROM User u JOIN u.room r WHERE r.room_uuid = :room_uuid")
boolean checkUsersByRoom_uuid(@Param("room_uuid") UUID room_uuid);
}
UserService.java
@RequiredArgsConstructor
@Service
public class UserService {
private final UserRepository userRepository;
private final StudyRoomRepository studyRoomRepository;
private final UserMapper userMapper;
//...
public GetUserResponse getUser(UUID user_uuid) {
User user =
userRepository.findUserByUserUuid(user_uuid).orElseThrow(NotFoundUser::new);
return userMapper.toDto(user);
}
//...
}
Video Chat
The signaling server for WebRTC uses WebSocket, so this needs to be set up.
WebSocketConfig.java
@EnableWebSocket
@RequiredArgsConstructor
@Configuration
public class WebSocketConfig implements WebSocketConfigurer {
private final SignalHandler signalHandler;
@Override
public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
registry.addHandler(signalHandler, "/signal").setAllowedOrigins("*"); // allow all origins
}
@Bean
public ServletServerContainerFactoryBean createWebSocketContainer() {
ServletServerContainerFactoryBean container = new ServletServerContainerFactoryBean();
container.setMaxTextMessageBufferSize(8192);
container.setMaxBinaryMessageBufferSize(8192);
return container;
}
}
Now let's implement a signalHandler for signaling. First, it's logic when connected to WebSocket and when diconnected.
SignalHanlder.java
@Component
@Slf4j
@RequiredArgsConstructor
public class SignalHandler extends TextWebSocketHandler {
private final UserRepository userRepository;
private final ObjectMapper objectMapper = new ObjectMapper();
private static final UUID SERVER_UUID = UUID.randomUUID();
private Map<UUID, WebSocketSession> userSessions = new HashMap<>();
// SDP Offer message
private static final String MSG_TYPE_OFFER = "offer";
// SDP Answer message
private static final String MSG_TYPE_ANSWER = "answer";
// New ICE Candidate message
private static final String MSG_TYPE_ICE = "ice";
// join room data message
private static final String MSG_TYPE_JOIN = "join";
// leave room data message
private static final String MSG_TYPE_LEAVE = "leave";
@Override
public void afterConnectionEstablished(final WebSocketSession session) {
// webSocket has been opened, send a message to the client
// when data field contains 'true' value, the client starts negotiating
// to establish peer-to-peer connection, otherwise they wait for a counterpart
log.debug("[ws] Session has been Open [session : {}]", session);
sendMessage(session, WebSocketMessage.builder().from(SERVER_UUID).type(MSG_TYPE_JOIN).build());
}
@Override
public void afterConnectionClosed(final WebSocketSession session, final CloseStatus status) {
log.debug("[ws] Session has been closed with status [session : {}, status : {}]", session, status);
log.debug("[ws] before userSessions : {}",userSessions);
UUID user_uuid = null;
for(UUID key : userSessions.keySet()) {
if(userSessions.get(key).equals(session)) {
user_uuid = key;
userSessions.remove(key);
break;
}
}
log.debug("[ws] after userSessions : {}, user_uuid : {}",userSessions,user_uuid);
userRepository.deleteByUser_uuid(user_uuid);
}
//...wait
}userSessions is a map for managing Websocket session for each user. In the signalHandler we refereed to, users(Websocket Session) were managed for each room object. However, since we manage the relationship between room and user in the database, SignalHanlder only needs to manage the WebSocketSession management.
The afterConnectionEstablished method sends a message to the client to join as an event that runs when the client connects to Websocket.
And the afterConnectionCloed method is an event that runs when the connection between the user and Websocket is disconnected. In this method, the data and Websocket session of the disconnected user are found in the database and deleted.
SignalHandler.java
public class SignalHandler extends TextWebSocketHandler {
//...
@Override
protected void handleTextMessage(final WebSocketSession session, final TextMessage textMessage) {
// a message has been received
try {
WebSocketMessage message =
objectMapper.readValue(textMessage.getPayload(), WebSocketMessage.class);
log.debug("[ws] Message of {} type from {} received", message.getType(), message.getFrom());
log.debug("[ws] Message : {}",message);
switch (message.getType()) {
case MSG_TYPE_OFFER:
case MSG_TYPE_ANSWER:
case MSG_TYPE_ICE:
Object candidate = message.getCandidate();
Object sdp = message.getSdp();
log.debug(
"[ws] Signal: {}",
candidate != null
? candidate.toString().substring(0, 64)
: sdp.toString().substring(0, 64));
List<User> users = userRepository.findAllByRoom_uuid(message.getData());
log.info("[ws] users : {}", users.toString());
for (User user : users) {
if (!message.getFrom().equals(user.getUser_uuid())) {
sendMessage(
userSessions.get(user.getUser_uuid()),
WebSocketMessage.builder()
.from(message.getFrom())
.type(message.getType())
.data(message.getData())
.sdp(message.getSdp())
.candidate(message.getCandidate())
.build());
}
}
break;
// identify user and their opponent
case MSG_TYPE_JOIN:
// message.data contains connected room id
log.debug("[ws] {} has joined Room: #{}", message.getFrom(), message.getData());
userSessions.put(message.getFrom(), session);
break;
case MSG_TYPE_LEAVE:
// message data contains connected room id
log.debug("[ws] {} is going to leave Room: #{}", message.getFrom(),message.getData());
break;
// something should be wrong wit
default:
log.debug("[ws] Type of the received message {} is undefined!", message.getType());
// handle this if needed
}
} catch (IOException e) {
log.debug("An error occured: {}", e.getMessage());
}
}
private void sendMessage(WebSocketSession session, WebSocketMessage message) {
try {
String json = objectMapper.writeValueAsString(message);
log.info("[ws] sendMessage , json : {}", json);
session.sendMessage(new TextMessage(json));
} catch (IOException e) {
log.error("An error occured: {}", e.getMessage());
}
}
}- Offer, Answer, ICE
- after obtaining the information of users in the study room based on room_uuid(data), the information of the user(session, room_uuid, message.type, room_uuid, sdp) is sent to other users except for the user who sent the message.
- Join
- join is executed when the user joins the room, and stores the uuid and session in the userSessions.
WebSocketMessage configured as follows.
WebSocket.java
@ToString
@Builder
@Getter
@NoArgsConstructor
@AllArgsConstructor
public class WebSocketMessage {
private UUID from;
private String type;
private UUID data;
private Object candidate;
private Object sdp;
}
Now let's look at the front.
chat_room.html
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:th="http://www.thymeleaf.org">
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Chat Room</title>
<!-- Latest minified Bootstrap & JQuery-->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.1.0/css/bootstrap.min.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.1.0/js/bootstrap.min.js"></script>
<link rel="stylesheet" type="text/css" href="/css/main.css"/>
</head>
<body class="text-center">
<!-- Begin page content -->
<main role="main" class="container-fluid">
<h1>Simple WebRTC Signalling Server</h1>
<input type="hidden" id="room_uuid" name="room_uuid" th:value="${studyRoom.room_uuid}"/>
<input type="hidden" id="has_user" name="has_user" th:value="${studyRoom.hasUser}"/>
<input type="hidden" id="user_uuid" name="user_uuid" th:value="${user.user_uuid}"/>
<div class="col-lg-12 mb-3">
<div class="mb-3" th:text="'User: ' + ${user.user_name} + ' @ Room #' + ${studyRoom.room_name}">
Local User Id
</div>
<div class="col-lg-12 mb-3">
<div class="d-flex justify-content-around mb-3">
<div id="buttons" class="row">
<div class="btn-group mr-2" role="group">
<div class="mr-2" data-toggle="buttons">
<label class="btn btn-outline-success" id="video_off">
<input type="radio" name="options" style="display:none" autocomplete="off">Video On
</label>
<label class="btn btn-outline-warning active" id="video_on">
<input type="radio" name="options" style="display:none" autocomplete="off" checked>Video Off
</label>
</div>
<div class="mr-2" data-toggle="buttons">
<label class="btn btn-outline-success" id="audio_off">
<input type="radio" name="options" style="display:none" autocomplete="off">Audio On
</label>
<label class="btn btn-outline-warning active" id="audio_on">
<input type="radio" name="options" style="display:none" autocomplete="off" checked>Audio Off
</label>
</div>
</div>
<!--<button type="button" class="btn btn-outline-success" id="audio" data-toggle="button">Audio</button>-->
<a th:href="@{/room/{id}/user/{uuid}/exit(id=${user.user_uuid},uuid=${studyRoom.room_uuid})}">
<button type="button" class="btn btn-outline-danger" id="exit" name="exit">
Exit Room
</button>
</a>
</div>
</div>
</div>
<div class="row justify-content-around mb-3">
<div class="col-lg-6 mb-3">
<video id="local_video" autoplay playsinline></video>
</div>
<div class="col-lg-6 mb-3">
<video id="remote_video" autoplay playsinline></video>
</div>
</div>
</div>
</main>
<script src="/js/webrtc_client.js"></script>
</body>
</html>
webrtc_client.js
'use strict';
// create and run Web Socket connection
const socket = new WebSocket("ws://" + window.location.host + "/signal");
// UI elements
const videoButtonOff = document.querySelector('#video_off');
const videoButtonOn = document.querySelector('#video_on');
const audioButtonOff = document.querySelector('#audio_off');
const audioButtonOn = document.querySelector('#audio_on');
const exitButton = document.querySelector('#exit');
const localRoom = document.querySelector('input#room_uuid').value;
const localVideo = document.getElementById('local_video');
const remoteVideo = document.getElementById('remote_video');
const hasUser = document.querySelector('input#has_user').value;
const localUserName = document.querySelector('input#user_uuid').value;
// WebRTC STUN servers
const peerConnectionConfig = {
'iceServers': [
{'urls': 'stun:stun.stunprotocol.org:3478'},
{'urls': 'stun:stun.l.google.com:19302'},
]
};
// WebRTC media
const mediaConstraints = {
audio: true,
video: true
};
// WebRTC variables
let localStream;
let localVideoTracks;
let myPeerConnection;
// on page load runner
$(function(){
start();
});
function start() {
// add an event listener for a message being received
socket.onmessage = function(msg) {
let message = JSON.parse(msg.data);
switch (message.type) {
case "text":
log('Text message from ' + message.from + ' received: ' + message.data);
break;
case "offer":
log('Signal OFFER received');
handleOfferMessage(message);
break;
case "answer":
log('Signal ANSWER received');
handleAnswerMessage(message);
break;
case "ice":
log('Signal ICE Candidate received');
handleNewICECandidateMessage(message);
break;
case "join":
log('Client is starting to '+(hasUser === 'true')?'negotiate' :'wait for a peer');
// log('Client is starting to ' + (message.data === "true)" ? 'negotiate' : 'wait for a peer'));
handlePeerConnection(message);
break;
default:
handleErrorMessage('Wrong type message received from server');
}
};
// add an event listener to get to know when a connection is open
socket.onopen = function() {
log('WebSocket connection opened to Room: #' + localRoom);
// send a message to the server to join selected room with Web Socket
sendToServer({
from: localUserName,
type: 'join',
data: localRoom
});
};
// a listener for the socket being closed event
socket.onclose = function(message) {
log('Socket has been closed');
};
// an event listener to handle socket errors
socket.onerror = function(message) {
handleErrorMessage("Error: " + message);
};
}
function stop() {
// send a message to the server to remove this client from the room clients list
log("Send 'leave' message to server");
sendToServer({
from: localUserName,
type: 'leave',
data: localRoom
});
if (myPeerConnection) {
log('Close the RTCPeerConnection');
// disconnect all our event listeners
myPeerConnection.onicecandidate = null;
myPeerConnection.ontrack = null;
myPeerConnection.onnegotiationneeded = null;
myPeerConnection.oniceconnectionstatechange = null;
myPeerConnection.onsignalingstatechange = null;
myPeerConnection.onicegatheringstatechange = null;
myPeerConnection.onnotificationneeded = null;
myPeerConnection.onremovetrack = null;
// Stop the videos
if (remoteVideo.srcObject) {
remoteVideo.srcObject.getTracks().forEach(track => track.stop());
}
if (localVideo.srcObject) {
localVideo.srcObject.getTracks().forEach(track => track.stop());
}
remoteVideo.src = null;
localVideo.src = null;
// close the peer connection
myPeerConnection.close();
myPeerConnection = null;
log('Close the socket');
if (socket != null) {
socket.close();
}
}
}
/*
UI Handlers
*/
// mute video buttons handler
videoButtonOff.onclick = () => {
localVideoTracks = localStream.getVideoTracks();
localVideoTracks.forEach(track => localStream.removeTrack(track));
$(localVideo).css('display', 'none');
log('Video Off');
};
videoButtonOn.onclick = () => {
localVideoTracks.forEach(track => localStream.addTrack(track));
$(localVideo).css('display', 'inline');
log('Video On');
};
// mute audio buttons handler
audioButtonOff.onclick = () => {
localVideo.muted = true;
log('Audio Off');
};
audioButtonOn.onclick = () => {
localVideo.muted = false;
log('Audio On');
};
// room exit button handler
exitButton.onclick = () => {
stop();
};
function log(message) {
console.log(message);
}
function handleErrorMessage(message) {
console.error(message);
}
// use JSON format to send WebSocket message
function sendToServer(msg) {
let msgJSON = JSON.stringify(msg);
socket.send(msgJSON);
}
// initialize media stream
function getMedia(constraints) {
if (localStream) {
localStream.getTracks().forEach(track => {
track.stop();
});
}
navigator.mediaDevices.getUserMedia(constraints)
.then(getLocalMediaStream).catch(handleGetUserMediaError);
}
// create peer connection, get media, start negotiating when second participant appears
function handlePeerConnection(message) {
createPeerConnection();
getMedia(mediaConstraints);
console.log("handlePeerConnection hasUser :",hasUser);
if (hasUser === 'true') {
myPeerConnection.onnegotiationneeded = handleNegotiationNeededEvent;
}
}
function createPeerConnection() {
myPeerConnection = new RTCPeerConnection(peerConnectionConfig);
// event handlers for the ICE negotiation process
myPeerConnection.onicecandidate = handleICECandidateEvent;
myPeerConnection.ontrack = handleTrackEvent;
// the following events are optional and could be realized later if needed
// myPeerConnection.onremovetrack = handleRemoveTrackEvent;
// myPeerConnection.oniceconnectionstatechange = handleICEConnectionStateChangeEvent;
// myPeerConnection.onicegatheringstatechange = handleICEGatheringStateChangeEvent;
// myPeerConnection.onsignalingstatechange = handleSignalingStateChangeEvent;
}
// add MediaStream to local video element and to the Peer
function getLocalMediaStream(mediaStream) {
localStream = mediaStream;
localVideo.srcObject = mediaStream;
localStream.getTracks().forEach(track => myPeerConnection.addTrack(track, localStream));
}
// handle get media error
function handleGetUserMediaError(error) {
log('navigator.getUserMedia error: ', error);
switch(error.name) {
case "NotFoundError":
alert("Unable to open your call because no camera and/or microphone were found.");
break;
case "SecurityError":
case "PermissionDeniedError":
// Do nothing; this is the same as the user canceling the call.
break;
default:
alert("Error opening your camera and/or microphone: " + error.message);
break;
}
stop();
}
// send ICE candidate to the peer through the server
function handleICECandidateEvent(event) {
if (event.candidate) {
sendToServer({
from: localUserName,
type: 'ice',
data:localRoom,
candidate: event.candidate
});
log('ICE Candidate Event: ICE candidate sent');
}
}
function handleTrackEvent(event) {
log('Track Event: set stream to remote video element');
remoteVideo.srcObject = event.streams[0];
}
// WebRTC called handler to begin ICE negotiation
// 1. create a WebRTC offer
// 2. set local media description
// 3. send the description as an offer on media format, resolution, etc
function handleNegotiationNeededEvent() {
myPeerConnection.createOffer().then(function(offer) {
return myPeerConnection.setLocalDescription(offer);
})
.then(function() {
sendToServer({
from: localUserName,
type: 'offer',
data: localRoom,
sdp: myPeerConnection.localDescription
});
log('Negotiation Needed Event: SDP offer sent');
})
.catch(function(reason) {
// an error occurred, so handle the failure to connect
handleErrorMessage('failure to connect error: ', reason);
});
}
function handleOfferMessage(message) {
log('Accepting Offer Message');
log(message);
let desc = new RTCSessionDescription(message.sdp);
//TODO test this
if (desc != null && message.sdp != null) {
log('RTC Signalling state: ' + myPeerConnection.signalingState);
myPeerConnection.setRemoteDescription(desc).then(function () {
log("Set up local media stream");
return navigator.mediaDevices.getUserMedia(mediaConstraints);
})
.then(function (stream) {
log("-- Local video stream obtained");
localStream = stream;
try {
localVideo.srcObject = localStream;
} catch (error) {
localVideo.src = window.URL.createObjectURL(stream);
}
log("-- Adding stream to the RTCPeerConnection");
localStream.getTracks().forEach(track => myPeerConnection.addTrack(track, localStream));
})
.then(function () {
log("-- Creating answer");
// Now that we've successfully set the remote description, we need to
// start our stream up locally then create an SDP answer. This SDP
// data describes the local end of our call, including the codec
// information, options agreed upon, and so forth.
return myPeerConnection.createAnswer();
})
.then(function (answer) {
log("-- Setting local description after creating answer");
// We now have our answer, so establish that as the local description.
// This actually configures our end of the call to match the settings
// specified in the SDP.
return myPeerConnection.setLocalDescription(answer);
})
.then(function () {
log("Sending answer packet back to other peer");
sendToServer({
from: localUserName,
data: localRoom,
type: 'answer',
sdp: myPeerConnection.localDescription
});
})
// .catch(handleGetUserMediaError);
.catch(handleErrorMessage)
}
}
function handleAnswerMessage(message) {
log("The peer has accepted request");
// Configure the remote description, which is the SDP payload
// in our "video-answer" message.
// myPeerConnection.setRemoteDescription(new RTCSessionDescription(message.sdp)).catch(handleErrorMessage);
myPeerConnection.setRemoteDescription(message.sdp).catch(handleErrorMessage);
}
function handleNewICECandidateMessage(message) {
let candidate = new RTCIceCandidate(message.candidate);
log("Adding received ICE candidate: " + JSON.stringify(candidate));
myPeerConnection.addIceCandidate(candidate).catch(handleErrorMessage);
}It's long, but let's find out one by one
socket.onopen = function() {
log('WebSocket connection opened to Room: #' + localRoom);
// send a message to the server to join selected room with Web Socket
sendToServer({
from: localUserName,
type: 'join',
data: localRoom
});
};when connected to Websocket, send user uuid and study room uuid with join to join type to signaling server. Then, in the signaling server, the user's session is stored in the userSessions. And at the same time, the signaling server sends a join type msessage.
const peerConnectionConfig = {
'iceServers': [
{'urls': 'stun:stun.stunprotocol.org:3478'},
{'urls': 'stun:stun.l.google.com:19302'},
]
};
function start() {
// add an event listener for a message being received
socket.onmessage = function(msg) {
let message = JSON.parse(msg.data);
switch (message.type) {
//...
case "join":
log('Client is starting to '+(hasUser === 'true')?'negotiate' :'wait for a peer');
handlePeerConnection(message);
break;
//...
}
};
}
//...
function handlePeerConnection(message) {
createPeerConnection();
getMedia(mediaConstraints);
console.log("handlePeerConnection hasUser :",hasUser);
if (hasUser === 'true') {
myPeerConnection.onnegotiationneeded = handleNegotiationNeededEvent;
}
}
function createPeerConnection() {
myPeerConnection = new RTCPeerConnection(peerConnectionConfig);
// event handlers for the ICE negotiation process
myPeerConnection.onicecandidate = handleICECandidateEvent;
myPeerConnection.ontrack = handleTrackEvent;
}
function handleICECandidateEvent(event) {
if (event.candidate) {
sendToServer({
from: localUserName,
type: 'ice',
data:localRoom,
candidate: event.candidate
});
log('ICE Candidate Event: ICE candidate sent');
}
}Since the signaling server sent a join type message, it executes the handlPeerConnection method. In handlePeerConnection, PeerConnection creation and Local media(video, voice) are set. Stun server uses stunprotocol and google.
If hasUser is true, so if there is another user in study room, handleNegotiationNeedEvent is excuted.
function handleNegotiationNeededEvent() {
myPeerConnection.createOffer().then(function(offer) {
return myPeerConnection.setLocalDescription(offer);
})
.then(function() {
sendToServer({
from: localUserName,
type: 'offer',
data: localRoom,
sdp: myPeerConnection.localDescription
});
log('Negotiation Needed Event: SDP offer sent');
})
.catch(function(reason) {
// an error occurred, so handle the failure to connect
handleErrorMessage('failure to connect error: ', reason);
});
}When the signaling server recieves an offer type message, it delivers the message to the offer type to other users. It also sends an ice type message, which is also delivered to other users.
function start() {
// add an event listener for a message being received
socket.onmessage = function(msg) {
let message = JSON.parse(msg.data);
switch (message.type) {
//...
case "offer":
log('Signal OFFER received');
handleOfferMessage(message);
break;
//...
}
};
}
function handleOfferMessage(message) {
log('Accepting Offer Message');
log(message);
let desc = new RTCSessionDescription(message.sdp);
//TODO test this
if (desc != null && message.sdp != null) {
log('RTC Signalling state: ' + myPeerConnection.signalingState);
myPeerConnection.setRemoteDescription(desc).then(function () {
log("Set up local media stream");
return navigator.mediaDevices.getUserMedia(mediaConstraints);
})
.then(function (stream) {
log("-- Local video stream obtained");
localStream = stream;
try {
localVideo.srcObject = localStream;
} catch (error) {
localVideo.src = window.URL.createObjectURL(stream);
}
log("-- Adding stream to the RTCPeerConnection");
localStream.getTracks().forEach(track => myPeerConnection.addTrack(track, localStream));
})
.then(function () {
log("-- Creating answer");
// Now that we've successfully set the remote description, we need to
// start our stream up locally then create an SDP answer. This SDP
// data describes the local end of our call, including the codec
// information, options agreed upon, and so forth.
return myPeerConnection.createAnswer();
})
.then(function (answer) {
log("-- Setting local description after creating answer");
// We now have our answer, so establish that as the local description.
// This actually configures our end of the call to match the settings
// specified in the SDP.
return myPeerConnection.setLocalDescription(answer);
})
.then(function () {
log("Sending answer packet back to other peer");
sendToServer({
from: localUserName,
data: localRoom,
type: 'answer',
sdp: myPeerConnection.localDescription
});
})
// .catch(handleGetUserMediaError);
.catch(handleErrorMessage)
}
}When another user receives an offer type message, it excutes the handleOfferMessage method. In this method, after setting up the media and remote of the user who sent the message, the answer type message is sent, including its own ifromation and sdp.
function start() {
// add an event listener for a message being received
socket.onmessage = function(msg) {
let message = JSON.parse(msg.data);
switch (message.type) {
//...
case "answer":
log('Signal ANSWER received');
handleAnswerMessage(message);
break;
//...
}
};
}
function handleAnswerMessage(message) {
log("The peer has accepted request");
// Configure the remote description, which is the SDP payload
// in our "video-answer" message.
// myPeerConnection.setRemoteDescription(new RTCSessionDescription(message.sdp)).catch(handleErrorMessage);
myPeerConnection.setRemoteDescription(message.sdp).catch(handleErrorMessage);
}The user who received the answer type message is done by remote setting.
Demonstration
Now let's see how it actually works.
In fact, I wanted to explain it better, but I feel sorry that I couldn't explain ti properly because of my lack of ability and vast infromation. I will try to explain it better next time.
[reference]
'Project' 카테고리의 다른 글
| Project_온라인스터디_Chpater0?_Persistence Context(N+1 Test) (0) | 2023.06.30 |
|---|---|
| Project_온라인스터디_Chapter0?_UniqueConstraint (0) | 2023.06.28 |
| Project_화상스터디_Chapter02_WebSocket (0) | 2023.04.01 |
| Project_화상스터디_Chapter01_WebRTC (0) | 2023.03.25 |
| Project_화상스터디_Chapter00_개요 & WebRTC (0) | 2023.03.24 |