customer-engineering/gitex2026

History

administrator db41397368 chore: auto-commit 2026-04-24 12:36		2026-04-24 12:36:21 +00:00
..
config.yaml	chore: auto-commit 2026-04-24 12:36	2026-04-24 12:36:21 +00:00
domain-scan	chore: auto-commit 2026-04-24 12:36	2026-04-24 12:36:21 +00:00
httpx	chore: auto-commit 2026-04-24 12:36	2026-04-24 12:36:21 +00:00
README.md	chore: auto-commit 2026-04-24 12:36	2026-04-24 12:36:21 +00:00
subfinder	chore: auto-commit 2026-04-24 12:36	2026-04-24 12:36:21 +00:00

README.md

Subdomain Finder

A comprehensive Go-based tool for discovering and verifying active subdomains through multiple techniques including passive enumeration, TLS certificate analysis, and HTTP service verification.

Built on Amazing Tools

This project stands on the shoulders of giants and integrates the excellent work from ProjectDiscovery:

subfinder - Fast passive subdomain enumeration tool
httpx - Fast and multi-purpose HTTP toolkit

Why This Project Exists

While subfinder and httpx are incredibly powerful tools on their own, this project provides:

Unified Workflow: Combines subdomain discovery, certificate analysis, and HTTP verification in one command
Keyword-Based Filtering: Automatically extracts organization keywords from domains to filter SSL certificate SANs
Structured Output: Provides consistent JSON/YAML output with detailed statistics and metadata
Library Integration: Offers a Go library for programmatic use in other security tools
Queue-Based Processing: Uses message queues for efficient concurrent domain processing

TL;DR: This is a wrapper that orchestrates subfinder and httpx with additional logic for enterprise security workflows.

Quick Start

# Download and install
curl -sSL https://github.com/valllabh/domain-scan/releases/latest/download/domain-scan_$(uname -s)_$(uname -m).tar.gz | tar -xz
sudo mv domain-scan /usr/local/bin/

# Run a basic scan
domain-scan discover example.com

# Or with Homebrew
brew install valllabh/tap/domain-scan
domain-scan discover example.com

# For development (from source)
git clone https://github.com/valllabh/domain-scan.git
cd domain-scan
make init && make run-discover

Overview

This tool integrates ProjectDiscovery's security tools to perform comprehensive subdomain discovery and verification:

Passive Discovery: Uses subfinder to enumerate subdomains from passive sources
TLS Certificate Analysis: Probes domains using httpx with TLS certificate inspection
HTTP Service Verification: Scans discovered subdomains for active HTTP/HTTPS services
Keyword Extraction: Automatically extracts keywords from domain names
Keyword Filtering: Filters domains from SSL certificates based on organizational relevance
Deduplication: Outputs unique active HTTP services

How It Works

The tool combines three powerful techniques for comprehensive subdomain discovery and verification:

1. Subfinder Integration

Uses ProjectDiscovery's subfinder to query passive DNS sources
Discovers subdomains from certificate transparency logs, DNS records, and other sources
Provides comprehensive initial subdomain enumeration without any active scanning

2. TLS Certificate Analysis

Inspects Subject Alternative Names (SANs) in SSL/TLS certificates
Finds additional subdomains not discovered by passive sources
Filters domains based on organizational relevance using keywords
Leverages certificate transparency for passive reconnaissance

SSL Certificate Keyword Filtering

When analyzing SSL certificates, domains often contain Subject Alternative Names (SANs) from multiple organizations due to shared hosting or third-party services. For example, if apple.com has a subdomain status.apple.com pointing to a third-party SaaS provider, that provider's certificate might also contain domains like status.microsoft.com or other unrelated organizations.

The keyword filtering system:

Extracts keywords from target domains (e.g., apple.com → apple, iphone.com → iphone)
Filters certificate domains to only include those matching organizational keywords
Prevents noise from unrelated domains in shared certificates
Examples:
- Target: apple.com → Keywords: apple
- Certificate contains: status.apple.com, store.apple.com, status.microsoft.com
- Filtered result: status.apple.com, store.apple.com (excludes status.microsoft.com)

3. HTTP Service Verification

Scans all discovered subdomains for active HTTP/HTTPS services
Tests multiple ports (configurable) for web services
Verifies actual accessibility and responsiveness
Returns only active, reachable services

Key Features

Integrated Subfinder: Built-in subfinder execution for comprehensive discovery
TLS Certificate Analysis: Inspects Subject Alternative Names in SSL certificates with keyword filtering
HTTP Service Scanning: Verifies active HTTP/HTTPS services on discovered subdomains
Configurable Port Scanning: Customizable port list for HTTP service detection
Automatic Keyword Extraction: Extracts keywords from domain names automatically
SSL Certificate Filtering: Filters domains from SSL certificates based on organizational relevance
Concurrent Processing: Uses httpx with configurable threads for fast scanning
Timeout Protection: Configurable timeouts for reliable operation
Progress Indicators: Real-time feedback on scanning progress
Deduplication: Automatically removes duplicate subdomains

Usage

domain-scan discover [domains...] [flags]

Basic Commands

# Get help
domain-scan --help
domain-scan discover --help

# Configuration management
domain-scan config show
domain-scan config init
domain-scan config set discovery.timeout 15

Available Options

Target and Keywords:

domains: Target domains for subdomain discovery
--keywords/-k: Additional keywords for filtering SSL certificate domains (auto-extracted from domains and combined with provided keywords)

Discovery Settings:

--timeout: Timeout in seconds (default: from config)
--threads: Number of concurrent threads (default: from config)

Output Options:

--output/-o: Output file path (default: stdout)
--format/-f: Output format (text, json)
--result-dir: Directory to save results (default: ./result)
--quiet/-q: Suppress progress output

Logging:

--loglevel: Log level (trace, debug, info, warn, error, silent)
--debug: Enable debug logging (deprecated, use --loglevel debug)

Configuration:

--config: Config file (default: $HOME/.domain-scan/config.yaml)

Examples

# Basic discovery (keywords automatically extracted from domain names)
domain-scan discover example.com

# Scan multiple domains
domain-scan discover example.com domain2.com

# Additional keywords (combined with auto-extracted ones)
domain-scan discover example.com --keywords staging,prod

# Custom discovery settings
domain-scan discover example.com --timeout 15 --threads 25

# Save results to JSON file
domain-scan discover example.com --output results.json --format json

# Custom result directory
domain-scan discover example.com --result-dir ./my-results

# Quiet mode with debug logging
domain-scan discover example.com --quiet --loglevel debug

# Multiple domains with custom settings
domain-scan discover example.com domain2.com --keywords api,admin --timeout 15

Configuration Management

The tool supports configuration files for persistent settings:

# Initialize default configuration
domain-scan config init

# View current configuration
domain-scan config show

# Set configuration values
domain-scan config set discovery.timeout 15
domain-scan config set keywords [mycompany,staging]

Default Ports: 80, 443, 8080, 8443, 3000, 8000, 8888 Configuration File: $HOME/.domain-scan/config.yaml

Integration with Main Project

This tool can be used standalone or integrated with the main reconnaissance script. The tool is self-contained and includes:

Subfinder Integration: Direct subfinder execution for comprehensive discovery
Automatic Keyword Extraction: No need for separate keyword extraction utilities
TLS Certificate Analysis: Additional discovery through certificate inspection with organizational filtering
HTTP Service Verification: Ensures only active services are reported
Unified Output: All active HTTP services in a single, deduplicated list

Built-in Security Tools

This tool integrates the amazing security tools from ProjectDiscovery:

subfinder - Fast passive subdomain enumeration tool
httpx - Fast and multi-purpose HTTP toolkit for probing and TLS inspection

These tools are now integrated directly into domain-scan, so no separate installation is required.

Configuration

The tool uses these default settings:

Timeout: 10 seconds per domain
Threads: 10 concurrent connections (TLS), 50 (HTTP scanning)
TLS Probe: Enabled for certificate inspection
Subfinder: Silent mode with all sources enabled
HTTP Scanning: Tests both HTTP and HTTPS protocols
Default Ports: 80,443,8080,8443,3000,8000,8888

Security Considerations

This tool is designed for defensive security purposes:

Performs passive reconnaissance and active HTTP probing
No exploitation or intrusive testing beyond HTTP requests
Helps organizations understand their external attack surface
Complies with responsible disclosure practices
Only tests for HTTP service availability

Output Format

The tool outputs active HTTP services to stdout, one per line, making it easy to pipe to other tools or save to files. Progress information and statistics are sent to stderr.

Example Output:

https://store.example.com
http://staging.example.com:8080
https://staging.example.com:8443

Progress Indicators

The tool provides real-time feedback through stderr:

🔍 Subdomain discovery progress
📋 Discovery statistics
🔐 TLS certificate analysis progress
🌐 HTTP service scanning progress
✅ Active service discoveries
📊 Final statistics

Workflow

Parse Arguments: Extract target domains, keywords, and ports
Keyword Extraction: Auto-extract keywords from domain names and combine with any manually provided keywords
Subfinder Discovery: Run subfinder to get initial subdomain list
TLS Certificate Analysis: Probe domains for additional subdomains via certificate SANs, filtering by organizational relevance
HTTP Service Scanning: Test all discovered subdomains for active HTTP services
SSL Certificate Filtering: Filter certificate domains based on keyword relevance to target organization
Deduplication: Remove duplicate entries
Output: Print active HTTP services to stdout

Limitations

TLS certificate analysis limited to domains with valid SSL/TLS certificates
SSL certificate keyword filtering may exclude domains from shared certificates that don't match organizational keywords
HTTP scanning limited to specified ports
Performance depends on target domain response times and network connectivity
Large subdomain lists may take considerable time to scan

Installation

Binary Downloads

Download the latest release for your platform from the releases page.

Package Managers

Homebrew (macOS/Linux)

brew install valllabh/tap/domain-scan

APT (Debian/Ubuntu)

wget https://github.com/valllabh/domain-scan/releases/latest/download/domain-scan_amd64.deb
sudo dpkg -i domain-scan_amd64.deb

RPM (RHEL/CentOS/Fedora)

wget https://github.com/valllabh/domain-scan/releases/latest/download/domain-scan_amd64.rpm
sudo rpm -i domain-scan_amd64.rpm

Alpine Linux

wget https://github.com/valllabh/domain-scan/releases/latest/download/domain-scan_amd64.apk
sudo apk add --allow-untrusted domain-scan_amd64.apk

From Source

go install github.com/valllabh/domain-scan@latest

Testing

The tool includes comprehensive test coverage:

# Run all tests
go test -v ./...

# Run with coverage
make test-coverage

# Run specific test packages
go test -v ./pkg/utils
go test -v ./pkg/discovery
go test -v ./pkg/domainscan

Development

This section covers everything needed for developing and contributing to domain-scan.

Quick Navigation:

Building
Running During Development
Code Quality
Make Commands Reference

Building

# Build for current platform
make build

# Build for all platforms
make build-all

# Development build with race detection
make dev

Running During Development

# Run with custom arguments
make run ARGS="discover example.com --keywords staging,prod"

# Quick shortcuts for testing
make run-help          # Show help
make run-discover      # Test discovery with example.com
make run-config        # Show current configuration

# Examples
make run ARGS="discover example.com --keywords staging,prod --ports 80,443"
make run ARGS="discover test.com --format json --output results.json"

Code Quality

# Format code
make fmt

# Lint code
make lint

# Security scan
make security

# Vulnerability check
make vuln

Make Commands Reference

The project includes a comprehensive Makefile with targets for all development tasks. Run make help to see all available targets.

Quick Start: make init && make test && make run-discover

Build Targets

make build           # Build for current platform
make build-all       # Build for multiple platforms (Linux, macOS)
make dev             # Development build with race detection
make clean           # Clean build artifacts

Development & Testing

make run ARGS="..."  # Build and run with custom arguments
make run-help        # Show application help
make run-discover    # Quick test discovery with example.com
make run-config      # Show current configuration

make test            # Run all tests
make test-coverage   # Run tests with HTML coverage report
make bench           # Run benchmark tests

Code Quality & Security

make fmt             # Format code with gofmt
make lint            # Run golangci-lint (installs if needed)
make security        # Run gosec security scanner
make vuln            # Check for vulnerabilities with govulncheck

Dependencies & Environment

make deps            # Install and verify Go dependencies
make init            # Initialize development environment
make update          # Update all dependencies

Release & Distribution

make release         # Create release using GoReleaser
make snapshot        # Create snapshot release for testing

Installation

make install         # Install binary to $GOPATH/bin
make uninstall       # Remove binary from $GOPATH/bin

Documentation & Help

make docs            # Generate command documentation
make help            # Show all available targets

Example Development Workflow

# Initialize environment
make init

# Run tests
make test

# Test the application
make run-discover

# Test with custom arguments
make run ARGS="discover example.com --keywords api,admin --ports 80,443,8080"

# Format and lint code
make fmt lint

# Create a snapshot build
make snapshot

Release Process

This project uses automated releases via GitHub Actions and GoReleaser.

Creating a Release

Ensure all changes are on the main branch
Create and push a new tag:
```
git tag v1.0.0
git push origin v1.0.0
```
GitHub Actions will automatically:
- Run all tests and quality checks
- Build binaries for multiple platforms (Linux, macOS)
- Create packages (DEB, RPM, APK)
- Generate release notes and publish to GitHub
- Update Homebrew formula

Release Artifacts

Each release automatically produces:

Cross-platform binaries for Linux and macOS (Intel/ARM)
Linux packages (DEB, RPM, APK) for easy installation
Homebrew formula for macOS and Linux users
Checksums for artifact verification

Versioning

The project follows Semantic Versioning:

v1.0.0 - Major release with breaking changes
v1.1.0 - Minor release with new features
v1.1.1 - Patch release with bug fixes

Development Releases

For testing unreleased changes:

# Create a snapshot build locally
make snapshot

# Or use the manual test workflow in GitHub Actions

Troubleshooting

Common Issues:

Check network connectivity for target domains
Verify domain names are correct and accessible
For large subdomain lists, consider using smaller port ranges
Monitor system resources during intensive scans

Installation Issues:

For package installations, ensure you have appropriate permissions
For Homebrew, run brew update if the formula isn't found

Security

This tool is designed for defensive security and authorized reconnaissance only. Please review our Security Policy for:

Responsible usage guidelines
Vulnerability reporting process
Security best practices
Dependency security information

⚠️ Important: Only use this tool on systems you own or have explicit permission to test.

Contributing

Contributions are welcome! Please:

Read our Security Policy first
Fork the repository
Create a feature branch
Make your changes with tests
Run make test and make lint
Submit a pull request

Acknowledgments

This project is built on the excellent work from the security community:

Core Dependencies

ProjectDiscovery - For creating amazing open source security tools
- subfinder - Fast passive subdomain enumeration tool
- httpx - Fast and multi-purpose HTTP toolkit
Cobra - CLI framework for Go
Viper - Configuration management for Go

Special Thanks

The ProjectDiscovery team for building the security tools that power this project
The Go security community for creating robust security tooling
All contributors who help improve this tool

💖 If you find this tool useful, please star the repositories of the amazing tools it depends on!

License

This project is licensed under the MIT License - see the LICENSE file for details.