GameStatsPx

A real-time game statistics processing platform with machine learning capabilities

GameStatsPx is a distributed real-time data processing platform designed to collect, process, and analyze game statistics using modern data engineering tools and machine learning techniques.

Features

Real-time Data Pipeline: Stream processing with Apache Kafka
Machine Learning: Linear and non-linear regression models using Apache Spark
Distributed Architecture: Microservices-based design with Docker
Data Storage: Elasticsearch for indexing and fast querying
Visualization: Kibana dashboards for real-time analytics
Log Processing: Fluent Bit for centralized log aggregation
Monitoring: Kafka UI for message queue monitoring
Scalable: Horizontal scaling capabilities at multiple layers
Security: X-Pack security with authentication and RBAC

Architecture

┌─────────────┐
│   Crawler   │ ──── Scrapes game data
└──────┬──────┘
       │ logs
       ▼
┌─────────────┐
│ Fluent Bit  │ ──── Aggregates & forwards
└──────┬──────┘
       │ stream
       ▼
┌─────────────┐
│    Kafka    │ ──── Message queue (Topic: gameStatsPx)
└──────┬──────┘
       │ consume
       ▼
┌─────────────┐
│    Spark    │ ──── ML predictions (Linear & Non-linear regression)
└──────┬──────┘
       │ results
       ▼
┌──────────────┐
│Elasticsearch │ ──── Storage & indexing
└──────┬───────┘
       │
       ▼
┌─────────────┐
│   Kibana    │ ──── Visualization
└─────────────┘

For a detailed architecture diagram, see architecture.

Prerequisites

Docker 20.10+
Docker Compose 1.29+
Python 3.8+
Git

Installation

1. Clone the Repository

git clone git@github.com:makapx/gameStatsPx.git
cd gameStatsPx

2. Set your IGDB token and client ID

cd gamges-crawler
mv config.example.json config.json

Set your credential into config.json file

Quick Start

1. Start All Services

docker-compose up -d

This command will start all services in detached mode:

Kafka (with KRaft)
Kafka UI
Fluent Bit
Elasticsearch
Kibana
Spark
Games Crawler

2. Verify Services Are Running

docker-compose ps

All services should show status as "Up".

3. Initialize Dataset

Before running the initialization script, you need to set up a Python virtual environment:

# Navigate to the games-crawler directory
cd games-crawler

# Create a virtual environment
python3 -m venv venv

# Activate the virtual environment
# On Linux/macOS:
source venv/bin/activate
# On Windows:
# venv\Scripts\activate

# Install required dependencies
pip install elasticsearch requests

# Run the initialization script
python dataset_gen.py

# Deactivate virtual environment when done
deactivate

# Return to project root
cd ..

4. Access Web Interfaces

Kafka UI: http://localhost:8585
Kibana: http://localhost:5601
Elasticsearch: http://localhost:9200

Default Credentials

Elasticsearch:

Username: elastic
Password: changeme

Kibana:

Username: kibana_system_user
Password: kibanapass123

Project Structure

gameStatsPx/
├── README.md
├── docker-compose.yml          # Main orchestration 
├── README.md                   # This file
├── games-crawler/              # Data collection 
│   ├── Dockerfile
│   ├── dataset_gen.py          # Dataset 
│   ├── games.py                # Crawler
├── spark/                      # Apache Spark ML 
│   ├── Dockerfile
├── fluent-bit/                 # Log processing 
│   └── fluent-bit.conf
└── docs/                       # Documentation
    ├── architecture.svg        # Architecture diagram
    ├── doc.ipynb               # Jupyter presentation

Configuration

Kafka Configuration

Kafka is configured in KRaft mode (no Zookeeper required).

Broker ID: 1
Topic: gameStatsPx (auto-created)
Partitions: 1
Replication Factor: 1

Elasticsearch Configuration

Cluster Mode: Single-node
Heap Size: 512MB (Xms/Xmx)
Security: X-Pack enabled
Port: 9200

Spark Configuration

The Spark service implements:

Linear Regression: For linear relationships in game statistics
Non-Linear Regression: For complex patterns and predictions

Output datasets are stored in ./spark/data/container_data/.

Usage

Starting Services

# Start all services
docker-compose up -d

# Start specific service
docker-compose up -d kafka

Stopping Services

# Stop all services
docker-compose down

# Stop and remove volumes (WARNING: deletes data)
docker-compose down -v

Viewing Logs

# View all logs
docker-compose logs -f

# View specific service logs
docker-compose logs -f spark
docker-compose logs -f kafka
docker-compose logs -f elasticsearch

Restarting Services

# Restart all services
docker-compose restart

# Restart specific service
docker-compose restart kafka

Rebuilding Containers

# Rebuild all containers
docker-compose build

# Rebuild specific container
docker-compose build crawler

# Rebuild and restart
docker-compose up -d --build

Documentation

Viewing the Jupyter Presentation

The project includes a comprehensive Jupyter notebook presentation.

# Navigate to docs directory
cd docs

# Create a virtual environment
python3 -m venv venv

# Activate the virtual environment
# On Linux/macOS:
source venv/bin/activate
# On Windows:
# venv\Scripts\activate

# Install Jupyter
pip install jupyter

# Start Jupyter
jupyter notebook doc.ipynb

# Deactivate when done
deactivate

Converting to Slideshow

You can convert the notebook to a presentation:

jupyter nbconvert doc.ipynb --to slides --post serve

Monitoring

Kafka Monitoring

Access Kafka UI at http://localhost:8585 to:

View topics and partitions
Monitor consumer groups
Check message throughput
Inspect message content

Elasticsearch Health

# Check cluster health
curl -u elastic:changeme http://localhost:9200/_cluster/health?pretty

# List indices
curl -u elastic:changeme http://localhost:9200/_cat/indices?v

# Get cluster stats
curl -u elastic:changeme http://localhost:9200/_cluster/stats?pretty

Kibana Dashboards

Navigate to http://localhost:5601
Log in with credentials
Go to Management → Stack Management → Index Patterns
Create index patterns for your data (games)
Use Discover and Dashboard to visualize data
Import games_dashboard.ndjson to Kibana > Saved Objects to see the dashboard I create

Troubleshooting

Services Not Starting

Check Docker:

docker --version
docker-compose --version

Check logs:

docker-compose logs [service-name]

Elasticsearch Connection Refused

Verify Elasticsearch is running:

docker-compose ps elasticsearch

Check health:

curl http://localhost:9200/_cluster/health

Kafka Connection Issues

Check if Kafka is running:

docker exec kafka kafka-topics --list --bootstrap-server localhost:39092

View Kafka logs:

docker-compose logs -f kafka

Python Script Errors

Ensure virtual environment is activated:

# In games-crawler directory
source venv/bin/activate  # Linux/macOS
# or
venv\Scripts\activate  # Windows

Verify dependencies:

pip list | grep -E "elasticsearch|requests"

Port Conflicts

If ports are already in use, modify docker-compose.yml:

ports:
  - "8585:8080"  # Change 8585 to another port

Memory Issues

If services crash due to memory:

Increase Docker memory:

Docker Desktop → Settings → Resources → Memory
Increase to at least 8GB

Reduce Elasticsearch heap:

environment:
  - "ES_JAVA_OPTS=-Xms256m -Xmx256m"  # Reduce from 512m

Name		Name	Last commit message	Last commit date
Latest commit History 10 Commits
docs		docs
fluent-bit		fluent-bit
games-crawler		games-crawler
spark		spark
.gitignore		.gitignore
README.md		README.md
docker-compose.yml		docker-compose.yml
games_dashboard.ndjson		games_dashboard.ndjson
games_dashboard_plain.ndjson		games_dashboard_plain.ndjson

Folders and files

Latest commit

History

Repository files navigation