Ep.30 Go and Log Analysis After Stress Testing - Enhancing Your System!

Go and Log Analysis After Stress Testing - Enhancing Your System!

In this episode, we will explore how to analyze logs and results from stress testing to identify bottlenecks in your WebSocket system, along with solutions to improve system performance!

Why Analyze Logs After Stress Testing?

Logs are crucial data sources that help us :

1. Understand issues that arise, such as dropped connections or failed message deliveries.
2. Identify overutilized resources, such as CPU, RAM, or network.
3. Plan system improvements to accommodate more users.

Types of Logs to Check

1. Connection Log : Check the number of successful and failed connections.

2. Latency Log : Analyze the time taken to respond to requests.

3. Error Log : Review errors, such as disconnections or undeliverable messages.

4. Resource Log : Monitor resource usage, such as CPU, RAM, and bandwidth.

Example of Log Analysis in Go

1. Increase Logging in WebSocket Server

In this example :

Log connections when new connections are made and when they end.

Log errors and messages received from users.

package main

import (
    "log"
    "net/http"

    "github.com/gorilla/websocket"
)

var upgrader = websocket.Upgrader{
    CheckOrigin: func(r *http.Request) bool {
        return true
    },
}

func handleConnections(w http.ResponseWriter, r *http.Request) {
    conn, err := upgrader.Upgrade(w, r, nil)
    if err != nil {
        log.Printf("Error upgrading connection: %v", err)
        return
    }
    defer conn.Close()

    log.Printf("New client connected: %s", r.RemoteAddr)

    for {
        _, msg, err := conn.ReadMessage()
        if err != nil {
            log.Printf("Error reading message from %s: %v", r.RemoteAddr, err)
            break
        }
        log.Printf("Received from %s: %s", r.RemoteAddr, msg)

        if err := conn.WriteMessage(websocket.TextMessage, msg); err != nil {
            log.Printf("Error writing message to %s: %v", r.RemoteAddr, err)
            break
        }
    }

    log.Printf("Client disconnected: %s", r.RemoteAddr)
}

func main() {
    http.HandleFunc("/ws", handleConnections)
    log.Println("WebSocket server started at :8080")
    log.Fatal(http.ListenAndServe(":8080", nil))
}

2. Use a Log Analyzer for Analysis

You can utilize tools like ELK Stack (Elasticsearch, Logstash, Kibana) to collect and analyze logs :

Elasticsearch : Stores log data.

Logstash : Sends data from the WebSocket server to Elasticsearch.

Kibana : Displays logs graphically.

Troubleshooting After Log Analysis

1. Failed Connection Issues

Causes :

• Server full (Connection Limit).
• High Network Latency.

Solutions :

• Increase the number of WebSocket servers in the cluster.
• Adjust the ulimit on the server.

ulimit -n 65535
 

2. High Latency

Causes :

• Time-consuming message processing.
• Server overloaded.

Solutions :

• Use a Worker Pool to distribute processing.
• Review and optimize code in resource-intensive areas, such as database access.

3. Message Sending Errors

Causes :

• Dropped connections.
• Sending messages to unresponsive clients.

Solutions :

• Increase timeout for message sending.

conn.SetWriteDeadline(time.Now().Add(10 * time.Second))

• Use ping/pong to check connection health.

conn.SetPingHandler(func(appData string) error {
    log.Println("Ping received")
    return conn.WriteControl(websocket.PongMessage, []byte(appData), time.Now().Add(10*time.Second))
})

4. Overutilized Resources

Causes :

• Excessive number of connections.
• Lack of load balancing.

Solutions :

• Use a Load Balancer like NGINX or Kubernetes.
• Check Redis Pub/Sub usage to reduce redundant processing.

Post-Improvement Review. After resolving issues, perform stress testing again and compare results :

• Has latency decreased?
• Has the number of successful connections increased?
• Has resource usage decreased?

Summary

• Analyze logs to troubleshoot issues like failed connections or high latency.
• Use tools like ELK Stack or Kibana for analysis.
• Enhance the system using load balancers, worker pools, and timeout settings.

Fun Activity

Try using the ELK Stack to analyze logs from your WebSocket server and create a graph showing average latency over time!