Building a Startup-Investment Algorithm in Python: End-to-End Tutorial with Crunchbase & YC Open Data
Building a Startup-Investment Algorithm in Python: End-to-End Tutorial with Crunchbase & YC Open Data
Introduction
The venture capital landscape has evolved dramatically, with global startup funding reaching $445 billion in 2023, highlighting the critical importance of identifying emerging market trends and investment opportunities. (Skylark AI) Traditional methods of trend identification in venture capital, such as manual research and network intelligence, are no longer sufficient due to the fast-paced market environment. (Skylark AI)
Artificial intelligence is reshaping how venture capital firms identify and analyze market trends, enabling them to spot emerging opportunities months or years before they become apparent to the broader market. (Skylark AI) This comprehensive tutorial will walk you through building a machine learning algorithm to predict startup success using Python, leveraging data from Crunchbase and Y Combinator's open datasets.
We'll cover everything from data extraction and feature engineering to model training and explainability, providing you with a complete framework for startup investment analysis. By the end of this tutorial, you'll have a working gradient boosting model that can predict Series A success rates, complete with SHAP explanations and Docker deployment files for easy replication.
Understanding the Data Landscape
Crunchbase: The Foundation of Startup Data
Crunchbase is a comprehensive public resource for financial information of various public and private companies and investments. (Medium - Crunchbase Scraping) The platform contains thousands of company profiles, which include investment data, funding information, leadership positions, mergers, news and industry trends. (Medium - Crunchbase Scraping)
As a major provider of private-company prospecting and research solutions, Crunchbase serves over 75 million individuals worldwide. (Crawlbase) In February 2024, Crunchbase attracted 7.7 million visitors, demonstrating its significance as a platform for entrepreneurs, investors, sales professionals, and market researchers, with a repository of data including 3 million listed companies. (Crawlbase)
The Technical Architecture
Crunchbase has recently converted its backend database to a Neo4j graph database. (Domino Data Lab) However, the data is exposed similarly to how it always has been: individual entities are retrieved and attribute data must be used to form edges between them prior to any graph analysis. (Domino Data Lab)
Setting Up Your Development Environment
Prerequisites and Dependencies
Before diving into the code, ensure you have the following Python packages installed:
# Core data manipulation and analysis
import pandas as pd
import numpy as np
from datetime import datetime, timedelta
# Machine learning libraries
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
# Feature engineering and selection
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.preprocessing import PolynomialFeatures
# Model explainability
import shap
# Data visualization
import matplotlib.pyplot as plt
import seaborn as sns
# Web scraping and API calls
import requests
import time
from bs4 import BeautifulSoup
import json
Docker Configuration
Create a Dockerfile for easy environment replication:
FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "startup_predictor.py"]
Data Extraction and Collection
Scraping Crunchbase Data
The tutorial uses the hidden web data web scraping approach using Python with an HTTP client library. (Medium - Crunchbase Scraping) Here's how to implement a robust data extraction pipeline:
class CrunchbaseExtractor:
def __init__(self, delay=2):
self.session = requests.Session()
self.delay = delay
self.headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}
self.session.headers.update(self.headers)
def extract_company_data(self, company_urls):
"""Extract comprehensive company data from Crunchbase URLs"""
companies_data = []
for url in company_urls:
try:
response = self.session.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
company_data = {
'name': self._extract_company_name(soup),
'founded_date': self._extract_founded_date(soup),
'total_funding': self._extract_total_funding(soup),
'funding_rounds': self._extract_funding_rounds(soup),
'employees': self._extract_employee_count(soup),
'industry': self._extract_industry(soup),
'location': self._extract_location(soup),
'founders': self._extract_founders(soup)
}
companies_data.append(company_data)
time.sleep(self.delay) # Rate limiting
except Exception as e:
print(f"Error extracting data for {url}: {e}")
continue
return pd.DataFrame(companies_data)
Y Combinator Data Integration
Existing projects like the 'ycprediction' repository demonstrate approaches to predict whether a company is going to Y Combinator or not. (GitHub - YC Prediction) We can build upon these foundations to create more sophisticated prediction models.
def load_yc_data():
"""Load and preprocess Y Combinator batch data"""
# YC provides batch data through their API
yc_companies = []
# Example structure for YC company data
yc_data = {
'company_name': [],
'batch': [],
'vertical': [],
'status': [], # Active, Acquired, IPO, Dead
'valuation': [],
'series_a_raised': []
}
return pd.DataFrame(yc_data)
Feature Engineering for Startup Success
Founder-Based Features
Recent research explores the use of large language models (LLMs) in venture capital decision-making, specifically in predicting startup success based on founder characteristics. (arXiv - Automating VC) LLM prompting techniques, such as chain-of-thought, are used to generate features from limited data. (arXiv - Automating VC)
class FounderFeatureExtractor:
def __init__(self):
self.education_rankings = self._load_university_rankings()
self.company_valuations = self._load_company_valuations()
def extract_founder_features(self, founders_data):
"""Extract quantitative features from founder backgrounds"""
features = {}
# Educational background
features['avg_education_rank'] = self._calculate_avg_education_rank(founders_data)
features['ivy_league_founders'] = self._count_ivy_league_founders(founders_data)
# Professional experience
features['avg_years_experience'] = self._calculate_avg_experience(founders_data)
features['big_tech_experience'] = self._count_big_tech_experience(founders_data)
features['previous_startup_experience'] = self._count_startup_experience(founders_data)
# Team composition
features['technical_founders'] = self._count_technical_founders(founders_data)
features['business_founders'] = self._count_business_founders(founders_data)
features['founder_team_size'] = len(founders_data)
return features
Market and Industry Features
Machine learning projects for startup success prediction typically use various features such as funding, location, industry, and team size. (GitHub - Startup Success Prediction) The goal is to help investors and entrepreneurs make more informed decisions about which startups to invest in or launch. (GitHub - Startup Success Prediction)
def engineer_market_features(df):
"""Create market-based features for startup success prediction"""
# Time-based features
df['days_since_founding'] = (datetime.now() - pd.to_datetime(df['founded_date'])).dt.days
df['founding_year'] = pd.to_datetime(df['founded_date']).dt.year
df['market_cycle'] = df['founding_year'].apply(classify_market_cycle)
# Funding velocity features
df['funding_per_year'] = df['total_funding'] / (df['days_since_founding'] / 365)
df['rounds_per_year'] = df['funding_rounds'] / (df['days_since_founding'] / 365)
# Location-based features
df['is_silicon_valley'] = df['location'].str.contains('San Francisco|Palo Alto|Mountain View')
df['is_tier1_city'] = df['location'].str.contains('New York|Boston|Seattle|Austin')
# Industry momentum features
industry_funding = df.groupby('industry')['total_funding'].agg(['mean', 'median', 'std'])
df = df.merge(industry_funding, left_on='industry', right_index=True, suffixes=('', '_industry'))
return df
Building the Prediction Model
Gradient Boosting Implementation
Early-stage startup investment is characterized by scarce data and uncertain outcomes. (arXiv - Policy Induction) Traditional machine learning approaches require large, labeled datasets and extensive fine-tuning, but remain opaque and difficult for domain experts to interpret or improve. (arXiv - Policy Induction)
class StartupSuccessPredictor:
def __init__(self):
self.model = GradientBoostingClassifier(random_state=42)
self.scaler = StandardScaler()
self.feature_selector = SelectKBest(f_classif, k=20)
self.label_encoders = {}
def prepare_features(self, df):
"""Prepare features for model training"""
# Define target variable (Series A success)
df['series_a_success'] = (df['funding_rounds'] >= 2) & (df['total_funding'] >= 5000000)
# Select relevant features
feature_columns = [
'days_since_founding', 'total_funding', 'funding_rounds',
'employees', 'avg_education_rank', 'technical_founders',
'funding_per_year', 'is_silicon_valley', 'is_tier1_city'
]
# Handle categorical variables
categorical_columns = ['industry', 'market_cycle']
for col in categorical_columns:
if col not in self.label_encoders:
self.label_encoders[col] = LabelEncoder()
df[f'{col}_encoded'] = self.label_encoders[col].fit_transform(df[col])
else:
df[f'{col}_encoded'] = self.label_encoders[col].transform(df[col])
feature_columns.append(f'{col}_encoded')
return df[feature_columns], df['series_a_success']
def train_model(self, X, y):
"""Train the gradient boosting model with hyperparameter tuning"""
# Split the data
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
# Scale features
X_train_scaled = self.scaler.fit_transform(X_train)
X_test_scaled = self.scaler.transform(X_test)
# Feature selection
X_train_selected = self.feature_selector.fit_transform(X_train_scaled, y_train)
X_test_selected = self.feature_selector.transform(X_test_scaled)
# Hyperparameter tuning
param_grid = {
'n_estimators': [100, 200, 300],
'max_depth': [3, 5, 7],
'learning_rate': [0.01, 0.1, 0.2],
'subsample': [0.8, 0.9, 1.0]
}
grid_search = GridSearchCV(
self.model, param_grid, cv=5, scoring='roc_auc', n_jobs=-1
)
grid_search.fit(X_train_selected, y_train)
self.model = grid_search.best_estimator_
# Evaluate model
y_pred = self.model.predict(X_test_selected)
y_pred_proba = self.model.predict_proba(X_test_selected)[:, 1]
print("Model Performance:")
print(f"ROC AUC Score: {roc_auc_score(y_test, y_pred_proba):.4f}")
print("\nClassification Report:")
print(classification_report(y_test, y_pred))
return X_test_selected, y_test, y_pred_proba
Advanced Model Interpretability with SHAP
Understanding Model Decisions
A lightweight ensemble framework that combines YES/NO questions generated by large language models (LLMs) can be used to predict startup success. (arXiv - Random Rule Forest) Each question generated by the LLM acts as a weak heuristic, which are then filtered, ranked, and aggregated through a threshold-based voting mechanism to construct a strong ensemble predictor. (arXiv - Random Rule Forest)
class ModelExplainer:
def __init__(self, model, X_train):
self.model = model
self.explainer = shap.TreeExplainer(model)
self.shap_values = self.explainer.shap_values(X_train)
def generate_explanations(self, X_test, feature_names):
"""Generate SHAP explanations for model predictions"""
shap_values_test = self.explainer.shap_values(X_test)
# Summary plot
plt.figure(figsize=(10, 8))
shap.summary_plot(shap_values_test, X_test, feature_names=feature_names, show=False)
plt.title('Feature Importance for Startup Success Prediction')
plt.tight_layout()
plt.savefig('shap_summary.png', dpi=300, bbox_inches='tight')
plt.show()
# Feature importance
feature_importance = np.abs(shap_values_test).mean(0)
importance_df = pd.DataFrame({
'feature': feature_names,
'importance': feature_importance
}).sort_values('importance', ascending=False)
return importance_df
def explain_individual_prediction(self, instance_idx, X_test, feature_names):
"""Explain individual startup prediction"""
shap_values_test = self.explainer.shap_values(X_test)
plt.figure(figsize=(10, 6))
shap.waterfall_plot(
shap.Explanation(
values=shap_values_test[instance_idx],
base_values=self.explainer.expected_value,
data=X_test[instance_idx],
feature_names=feature_names
)
)
plt.title(f'Prediction Explanation for Startup {instance_idx}')
plt.tight_layout()
plt.savefig(f'explanation_startup_{instance_idx}.png', dpi=300, bbox_inches='tight')
plt.show()
Model Validation and Performance Analysis
Benchmarking Against Existing Approaches
On a test set where 10% of startups are classified as successful, advanced approaches achieve a precision rate of 50%, representing a 5x improvement over random selection. (arXiv - Random Rule Forest) This benchmark provides a target for our model performance.
def validate_model_performance(model, X_test, y_test, y_pred_proba):
"""Comprehensive model validation and performance analysis"""
# ROC Curve Analysis
from sklearn.metrics import roc_curve, auc
fpr, tpr, thresholds = roc_curve(y_test, y_pred_proba)
roc_auc = auc(fpr, tpr)
plt.figure(figsize=(12, 5))
# ROC Curve
plt.subplot(1, 2, 1)
plt.plot(fpr, tpr, color='darkorange', lw=2, label=f'ROC curve (AUC = {roc_auc:.2f})')
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic')
plt.legend(loc="lower right")
# Precision-Recall Curve
from sklearn.metrics import precision_recall_curve, average_precision_score
precision, recall, _ = precision_recall_curve(y_test, y_pred_proba)
avg_precision = average_precision_score(y_test, y_pred_proba)
plt.subplot(1, 2, 2)
plt.plot(recall, precision, color='blue', lw=2, label=f'AP = {avg_precision:.2f}')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall Curve')
plt.legend()
plt.tight_layout()
plt.savefig('model_performance.png', dpi=300, bbox_inches='tight')
plt.show()
# Calculate precision at different thresholds
thresholds_to_test = [0.1, 0.2, 0.3, 0.4, 0.5]
performance_metrics = []
for threshold in thresholds_to_test:
y_pred_threshold = (y_pred_proba >= threshold).astype(int)
precision = precision_score(y_test, y_pred_threshold)
recall = recall_score(y_test, y_pred_threshold)
f1 = f1_score(y_test, y_pred_threshold)
performance_metrics.append({
'threshold': threshold,
'precision': precision,
'recall': recall,
'f1_score': f1
})
return pd.DataFrame(performance_metrics)
Production Deployment and Monitoring
Creating a Prediction Pipeline
Machine learning methods for predicting venture capital deals and returns require robust production systems. (GitHub - Venture AI) Here's how to create a deployable prediction pipeline:
class StartupPredictionPipeline:
def __init__(self, model_path='startup_model.pkl'):
self.model = None
self.scaler = None
self.feature_selector = None
self.label_encoders = {}
self.model_path = model_path
def save_model(self, model, scaler, feature_selector, label_encoders):
"""Save trained model and preprocessing components"""
import pickle
model_components = {
'model': model,
'scaler': scaler,
'feature_selector': feature_selector,
'label_encoders': label_encoders
}
with open(self.model_path, 'wb') as f:
pickle.dump(model_components, f)
def load_model(self):
"""Load trained model and preprocessing components"""
import pickle
with open(self.model_path, 'rb') as f:
components = pickle.load(f)
## Frequently Asked Questions
### What data sources are used in this startup investment algorithm tutorial?
This tutorial primarily uses Crunchbase data, which contains over 3 million company profiles with investment data, funding information, and industry trends, along with Y Combinator open data. Crunchbase serves over 75 million users worldwide and attracted 7.7 million visitors in February 2024, making it a comprehensive resource for startup analysis.
### How do you scrape Crunchbase data for the investment algorithm?
The tutorial demonstrates web scraping Crunchbase using Python with HTTP client libraries through a hidden web data approach. Since Crunchbase converted its backend to a Neo4j graph database, individual entities are retrieved and attribute data is used to form edges between them for network analysis and algorithm development.
### What machine learning techniques are effective for predicting startup success?
Recent research shows that memory-augmented large language models (LLMs) with in-context learning are effective for early-stage investment decisions where data is scarce. Traditional ML approaches require large labeled datasets, while newer methods like Random Rule Forest (RRF) achieve 50% precision rates, representing a 5x improvement over random selection.
### Why is AI important for venture capital investment decisions in 2024?
With global startup funding reaching $445 billion in 2023, traditional manual research methods are insufficient for the fast-paced market environment. AI enables venture capital firms to identify emerging market trends and opportunities months or years before they become apparent to the broader market, revolutionizing how VCs analyze potential investments.
### What features should be included in a startup success prediction model?
Effective startup prediction models should incorporate features such as funding amounts, geographic location, industry sector, team size, and founder characteristics. Research shows that LLM-powered feature engineering can extract insights from limited data using techniques like chain-of-thought prompting to generate meaningful predictive features.
### How accurate can startup investment algorithms be?
Modern startup prediction algorithms can achieve significant improvements over random selection. For example, ensemble methods using LLM-generated questions can achieve 50% precision rates on test sets where only 10% of startups are successful, representing a 5x improvement over random selection and providing substantial value for investment decision-making.
## Sources
1. [https://arxiv.org/abs/2407.04885](https://arxiv.org/abs/2407.04885)
2. [https://arxiv.org/abs/2505.21427](https://arxiv.org/abs/2505.21427)
3. [https://arxiv.org/abs/2505.24622](https://arxiv.org/abs/2505.24622)
4. [https://crawlbase.com/blog/how-to-scrape-crunchbase/](https://crawlbase.com/blog/how-to-scrape-crunchbase/)
5. [https://domino.ai/blog/crunchbase-network-analysis-with-python](https://domino.ai/blog/crunchbase-network-analysis-with-python)
6. [https://github.com/ankitkr0/ycprediction](https://github.com/ankitkr0/ycprediction)
7. [https://github.com/ifelifpass/venture.ai](https://github.com/ifelifpass/venture.ai)
8. [https://github.com/sumitjhaleriya/Startup-Success-Prediction-using-Machine-Learning](https://github.com/sumitjhaleriya/Startup-Success-Prediction-using-Machine-Learning)
9. [https://medium.com/@gpzzex/how-to-scrape-crunchbase-company-and-people-data-2024-update-a3fb73c00f72](https://medium.com/@gpzzex/how-to-scrape-crunchbase-company-and-people-data-2024-update-a3fb73c00f72)
10. [https://www.skylarkai.com/blog-page/revolutionizing-vc-with-ai](https://www.skylarkai.com/blog-page/revolutionizing-vc-with-ai)