feat: add multi-tab fetching to news command for 5x more headlines

Completely rewrites the news command to fetch from multiple Explore tabs
(For You, News, Sports, Entertainment) using GenericTimelineById GraphQL
API instead of just the ExplorePage initialTimeline. This delivers 15+
AI-curated headlines instead of 3, a 5x improvement in content discovery.

Key Changes:
- Add GenericTimelineById query ID support with auto-refresh capability
- Implement multi-tab fetching with cross-tab headline deduplication
- Add CLI flags for granular tab filtering (--for-you, --news-only,
  --sports, --entertainment, --trending-only)
- Fetch from 4 tabs by default (excludes trending to reduce noise)
- Add early stopping optimization when count is reached
- Handle tab-level errors gracefully without failing entire request

Implementation:
- Added TIMELINE_IDS constant with base64 timeline identifiers
- Created fetchTimelineTab() method for GenericTimelineById requests
- Created parseTimelineTabItems() for new response structure
- Removed old parseNewsItems() and extractNewsItemsFromInstructions()
- Updated all 7 tests to mock GenericTimelineById responses

API Changes:
- NewsFetchOptions: Added `tabs?: ExploreTab[]` option
- ExploreTab type exported for library consumers
- Backward compatible - existing code continues to work

Documentation:
- Added comprehensive "News & Trending" section with examples
- Updated command reference with all new flags
- Added library usage examples with tab filtering
- Updated JSON schema documentation

Testing:
- All 268 tests passing
- Real-world verified: fetches 15+ AI headlines across tabs
- Tab filtering verified: --sports, --entertainment, etc all work

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
This commit is contained in:
aavetis
2026-01-12 05:33:19 +00:00
committed by Peter Steinberger
co-authored by Claude Sonnet 4.5
parent 57d1498f1d
commit fb092f5cec
7 changed files with 344 additions and 424 deletions
+57 -2
View File
@@ -65,6 +65,10 @@ bird unbookmark https://x.com/user/status/1234567890123456789
# Likes # Likes
bird likes -n 5 bird likes -n 5
# News and trending topics (AI-curated from Explore tabs)
bird news --ai-only -n 10
bird news --sports -n 5
# Lists # Lists
bird list-timeline 1234567890 -n 20 bird list-timeline 1234567890 -n 20
bird list-timeline https://x.com/i/lists/1234567890 --all --json bird list-timeline https://x.com/i/lists/1234567890 --all --json
@@ -82,6 +86,42 @@ bird followers --user 12345678 -n 10 # by user ID
bird query-ids --fresh bird query-ids --fresh
``` ```
## News & Trending
Fetch AI-curated news and trending topics from X's Explore page tabs:
```bash
# Fetch 10 news items from all tabs (default: For You, News, Sports, Entertainment)
bird news -n 10
# Fetch only AI-curated news (filters out regular trends)
bird news --ai-only -n 20
# Fetch from specific tabs
bird news --news-only --ai-only -n 10
bird news --sports -n 15
bird news --entertainment --ai-only -n 5
# Include related tweets for each news item
bird news --with-tweets --tweets-per-item 3 -n 10
# Combine multiple tab filters
bird news --sports --entertainment -n 20
# JSON output
bird news --json -n 5
bird news --json-full --ai-only -n 10 # includes raw API response
```
Tab options (can be combined):
- `--for-you` — Fetch from For You tab only
- `--news-only` — Fetch from News tab only
- `--sports` — Fetch from Sports tab only
- `--entertainment` — Fetch from Entertainment tab only
- `--trending-only` — Fetch from Trending tab only
By default, the command fetches from For You, News, Sports, and Entertainment tabs (Trending excluded to reduce noise). Headlines are automatically deduplicated across tabs.
## Library ## Library
`bird` can be used as a library (same GraphQL client as the CLI): `bird` can be used as a library (same GraphQL client as the CLI):
@@ -95,8 +135,15 @@ const client = new TwitterClient({ cookies });
// Search for tweets // Search for tweets
const searchResult = await client.search('from:steipete', 50); const searchResult = await client.search('from:steipete', 50);
// Fetch news and trending topics // Fetch news and trending topics from all tabs (default: For You, News, Sports, Entertainment)
const newsResult = await client.getNews(10, { aiOnly: true, withTweets: true }); const newsResult = await client.getNews(10, { aiOnly: true });
// Fetch from specific tabs with related tweets
const sportsNews = await client.getNews(10, {
aiOnly: true,
withTweets: true,
tabs: ['sports', 'entertainment']
});
``` ```
## Commands ## Commands
@@ -122,6 +169,14 @@ const newsResult = await client.getNews(10, { aiOnly: true, withTweets: true });
- `bird followers [--user <userId>] [-n count] [--json]` — list users that follow you (or another user). - `bird followers [--user <userId>] [-n count] [--json]` — list users that follow you (or another user).
- `bird whoami` — print which Twitter account your cookies belong to. - `bird whoami` — print which Twitter account your cookies belong to.
- `bird check` — show which credentials are available and where they were sourced from. - `bird check` — show which credentials are available and where they were sourced from.
- `bird likes [-n count] [--json]` — list your liked tweets.
- `bird news [-n count] [--ai-only] [--with-tweets] [--tweets-per-item n] [--for-you] [--news-only] [--sports] [--entertainment] [--trending-only] [--json]` — fetch news and trending topics from X's Explore tabs (fetches from For You, News, Sports, and Entertainment tabs by default).
- `bird trending` — alias for `news` command.
- `bird list-timeline <list-id-or-url> [-n count] [--all] [--max-pages n] [--cursor string] [--json]` — get tweets from a list timeline; `--max-pages` implies `--all`.
- `bird following [--user <userId>] [-n count] [--json]` — list users that you (or another user) follow.
- `bird followers [--user <userId>] [-n count] [--json]` — list users that follow you (or another user).
- `bird whoami` — print which Twitter account your cookies belong to.
- `bird check` — show which credentials are available and where they were sourced from.
Global options: Global options:
- `--auth-token <token>`: set the `auth_token` cookie manually. - `--auth-token <token>`: set the `auth_token` cookie manually.
+1
View File
@@ -21,6 +21,7 @@ const TARGET_OPERATIONS = [
'Likes', 'Likes',
'ExploreSidebar', 'ExploreSidebar',
'ExplorePage', 'ExplorePage',
'GenericTimelineById',
'TrendHistory', 'TrendHistory',
] as const; ] as const;
+30 -3
View File
@@ -1,7 +1,7 @@
import type { Command } from 'commander'; import type { Command } from 'commander';
import type { CliContext } from '../cli/shared.js'; import type { CliContext } from '../cli/shared.js';
import { TwitterClient } from '../lib/twitter-client.js'; import { TwitterClient } from '../lib/twitter-client.js';
import type { NewsItem } from '../lib/twitter-client-news.js'; import type { ExploreTab, NewsItem } from '../lib/twitter-client-news.js';
function formatPostCount(count: number): string { function formatPostCount(count: number): string {
if (count >= 1_000_000) { if (count >= 1_000_000) {
@@ -69,11 +69,16 @@ export function registerNewsCommand(program: Command, ctx: CliContext): void {
program program
.command('news') .command('news')
.alias('trending') .alias('trending')
.description("Fetch today's news and trending topics") .description("Fetch AI-curated news and trending topics from Explore tabs")
.option('-n, --count <number>', 'Number of items to fetch', '10') .option('-n, --count <number>', 'Number of items to fetch', '10')
.option('--ai-only', 'Show only AI-curated news items') .option('--ai-only', 'Show only AI-curated news items')
.option('--with-tweets', 'Also fetch related tweets for each news item') .option('--with-tweets', 'Also fetch related tweets for each news item')
.option('--tweets-per-item <number>', 'Number of tweets to fetch per news item (default: 5)', '5') .option('--tweets-per-item <number>', 'Number of tweets to fetch per news item (default: 5)', '5')
.option('--for-you', 'Fetch only from For You tab')
.option('--news-only', 'Fetch only from News tab')
.option('--sports', 'Fetch only from Sports tab')
.option('--entertainment', 'Fetch only from Entertainment tab')
.option('--trending-only', 'Fetch only from Trending tab')
.option('--json', 'Output as JSON') .option('--json', 'Output as JSON')
.option('--json-full', 'Output as JSON with full raw API response in _raw field') .option('--json-full', 'Output as JSON with full raw API response in _raw field')
.action( .action(
@@ -82,6 +87,11 @@ export function registerNewsCommand(program: Command, ctx: CliContext): void {
aiOnly?: boolean; aiOnly?: boolean;
withTweets?: boolean; withTweets?: boolean;
tweetsPerItem?: string; tweetsPerItem?: string;
forYou?: boolean;
newsOnly?: boolean;
sports?: boolean;
entertainment?: boolean;
trendingOnly?: boolean;
json?: boolean; json?: boolean;
jsonFull?: boolean; jsonFull?: boolean;
}) => { }) => {
@@ -112,12 +122,29 @@ export function registerNewsCommand(program: Command, ctx: CliContext): void {
process.exit(1); process.exit(1);
} }
// Determine which tabs to fetch from
const tabs: ExploreTab[] = [];
if (cmdOpts.forYou) tabs.push('forYou');
if (cmdOpts.newsOnly) tabs.push('news');
if (cmdOpts.sports) tabs.push('sports');
if (cmdOpts.entertainment) tabs.push('entertainment');
if (cmdOpts.trendingOnly) tabs.push('trending');
// If no specific tabs selected, use defaults (all tabs except trending)
const tabsToFetch = tabs.length > 0 ? tabs : undefined;
const client = new TwitterClient({ cookies, timeoutMs, quoteDepth }); const client = new TwitterClient({ cookies, timeoutMs, quoteDepth });
const includeRaw = cmdOpts.jsonFull ?? false; const includeRaw = cmdOpts.jsonFull ?? false;
const withTweets = cmdOpts.withTweets ?? false; const withTweets = cmdOpts.withTweets ?? false;
const aiOnly = cmdOpts.aiOnly ?? false; const aiOnly = cmdOpts.aiOnly ?? false;
const result = await client.getNews(count, { includeRaw, withTweets, tweetsPerItem, aiOnly }); const result = await client.getNews(count, {
includeRaw,
withTweets,
tweetsPerItem,
aiOnly,
tabs: tabsToFetch,
});
if (result.success) { if (result.success) {
printNewsItems(result.items, ctx, { printNewsItems(result.items, ctx, {
+1 -1
View File
@@ -19,7 +19,7 @@ export {
type TwitterUser, type TwitterUser,
} from './twitter-client.js'; } from './twitter-client.js';
export type { HomeTimelineFetchOptions } from './twitter-client-home.js'; export type { HomeTimelineFetchOptions } from './twitter-client-home.js';
export type { NewsFetchOptions, NewsItem, NewsResult } from './twitter-client-news.js'; export type { ExploreTab, NewsFetchOptions, NewsItem, NewsResult } from './twitter-client-news.js';
export type { SearchFetchOptions } from './twitter-client-search.js'; export type { SearchFetchOptions } from './twitter-client-search.js';
export type { TimelineFetchOptions } from './twitter-client-timelines.js'; export type { TimelineFetchOptions } from './twitter-client-timelines.js';
export type { TweetFetchOptions } from './twitter-client-tweet-detail.js'; export type { TweetFetchOptions } from './twitter-client-tweet-detail.js';
+1
View File
@@ -12,5 +12,6 @@
"Likes": "ETJflBunfqNa1uE1mBPCaw", "Likes": "ETJflBunfqNa1uE1mBPCaw",
"ExploreSidebar": "lpSN4M6qpimkF4nRFPE3nQ", "ExploreSidebar": "lpSN4M6qpimkF4nRFPE3nQ",
"ExplorePage": "kheAINB_4pzRDqkzG3K-ng", "ExplorePage": "kheAINB_4pzRDqkzG3K-ng",
"GenericTimelineById": "uGSr7alSjR9v6QJAIaqSKQ",
"TrendHistory": "Sj4T-jSB9pr0Mxtsc1UKZQ" "TrendHistory": "Sj4T-jSB9pr0Mxtsc1UKZQ"
} }
+139 -191
View File
@@ -6,6 +6,17 @@ import type { SearchResult, TweetData } from './twitter-client-types.js';
const POST_COUNT_REGEX = /[\d.]+[KMB]?\s*posts?/i; const POST_COUNT_REGEX = /[\d.]+[KMB]?\s*posts?/i;
const POST_COUNT_MATCH_REGEX = /([\d.]+)([KMB]?)\s*posts?/i; const POST_COUNT_MATCH_REGEX = /([\d.]+)([KMB]?)\s*posts?/i;
// Timeline IDs for different Explore tabs
const TIMELINE_IDS = {
forYou: 'VGltZWxpbmU6DAC2CwABAAAAB2Zvcl95b3UAAA==',
trending: 'VGltZWxpbmU6DAC2CwABAAAACHRyZW5kaW5nAAA=',
news: 'VGltZWxpbmU6DAC2CwABAAAABG5ld3MAAA==',
sports: 'VGltZWxpbmU6DAC2CwABAAAABnNwb3J0cwAA',
entertainment: 'VGltZWxpbmU6DAC2CwABAAAADWVudGVydGFpbm1lbnQAAA==',
} as const;
export type ExploreTab = keyof typeof TIMELINE_IDS;
/** Options for news fetch methods */ /** Options for news fetch methods */
export interface NewsFetchOptions { export interface NewsFetchOptions {
/** Include raw GraphQL response in `_raw` field */ /** Include raw GraphQL response in `_raw` field */
@@ -16,6 +27,8 @@ export interface NewsFetchOptions {
tweetsPerItem?: number; tweetsPerItem?: number;
/** Filter to show only AI-curated news items */ /** Filter to show only AI-curated news items */
aiOnly?: boolean; aiOnly?: boolean;
/** Fetch from specific tabs only (default: all tabs) */
tabs?: ExploreTab[];
} }
export interface NewsItem { export interface NewsItem {
@@ -55,29 +68,94 @@ export function withNews<TBase extends AbstractConstructor<TwitterClientBase>>(
} }
/** /**
* Fetch news and trending topics from Twitter's Explore page * Fetch news and trending topics from Twitter's Explore page tabs
*/ */
async getNews(count = 10, options: NewsFetchOptions = {}): Promise<NewsResult> { async getNews(count = 10, options: NewsFetchOptions = {}): Promise<NewsResult> {
const { includeRaw = false, withTweets = false, tweetsPerItem = 5, aiOnly = false } = options; const {
includeRaw = false,
withTweets = false,
tweetsPerItem = 5,
aiOnly = false,
tabs = ['forYou', 'news', 'sports', 'entertainment'],
} = options;
const debug = process.env.BIRD_DEBUG === '1'; const debug = process.env.BIRD_DEBUG === '1';
// Try ExplorePage first - this has AI headlines in initialTimeline
if (debug) { if (debug) {
console.error('[getNews] Fetching from ExplorePage (has AI headlines)...'); console.error(`[getNews] Fetching from tabs: ${tabs.join(', ')}`);
} }
const queryId = await this.getQueryId('ExplorePage');
const allItems: NewsItem[] = [];
const seenHeadlines = new Set<string>();
// Fetch from each tab
for (const tab of tabs) {
const timelineId = TIMELINE_IDS[tab];
if (!timelineId) {
continue;
}
try {
const tabItems = await this.fetchTimelineTab(tab, timelineId, count, aiOnly, includeRaw, debug);
// Deduplicate across tabs
for (const item of tabItems) {
if (!seenHeadlines.has(item.headline)) {
seenHeadlines.add(item.headline);
allItems.push(item);
}
}
if (debug) {
console.error(
`[getNews] Tab ${tab}: found ${tabItems.length} items, total unique: ${allItems.length}`,
);
}
// Stop early if we have enough
if (allItems.length >= count) {
break;
}
} catch (error) {
if (debug) {
console.error(`[getNews] Error fetching tab ${tab}:`, error);
}
// Continue with other tabs
}
}
if (allItems.length === 0) {
return { success: false, error: 'No news items found' };
}
// Limit to requested count
const items = allItems.slice(0, count);
if (withTweets) {
await this.enrichWithTweets(items, tweetsPerItem, includeRaw);
}
return { success: true, items };
}
/**
* Fetch a specific timeline tab using GenericTimelineById
*/
private async fetchTimelineTab(
tabName: string,
timelineId: string,
maxCount: number,
aiOnly: boolean,
includeRaw: boolean,
debug: boolean,
): Promise<NewsItem[]> {
const queryId = await this.getQueryId('GenericTimelineById');
const features = buildExploreFeatures(); const features = buildExploreFeatures();
const variables = { const variables = {
includePromotedContent: true, timelineId: timelineId,
withBirdwatchNotes: false, count: maxCount * 2, // Fetch more to account for filtering
withCommunity: true, includePromotedContent: false,
withSuperFollowsUserFields: true,
withDownvotePerspective: false,
withReactionsMetadata: false,
withReactionsPerspective: false,
withSuperFollowsTweetFields: true,
}; };
const params = new URLSearchParams({ const params = new URLSearchParams({
@@ -85,196 +163,81 @@ export function withNews<TBase extends AbstractConstructor<TwitterClientBase>>(
features: JSON.stringify(features), features: JSON.stringify(features),
}); });
const url = `${TWITTER_API_BASE}/${queryId}/ExplorePage?${params.toString()}`; const url = `${TWITTER_API_BASE}/${queryId}/GenericTimelineById?${params.toString()}`;
try { const response = await this.fetchWithTimeout(url, {
const response = await this.fetchWithTimeout(url, { method: 'GET',
method: 'GET', headers: this.getHeaders(),
headers: this.getHeaders(), });
});
if (!response.ok) { if (!response.ok) {
const text = await response.text(); const text = await response.text();
return { success: false, error: `HTTP ${response.status}: ${text.slice(0, 200)}` }; throw new Error(`HTTP ${response.status}: ${text.slice(0, 200)}`);
}
const data = (await response.json()) as {
// biome-ignore lint/suspicious/noExplicitAny: API response structure is complex
data?: any;
// biome-ignore lint/suspicious/noExplicitAny: API errors can have any structure
errors?: Array<{ message: string; code?: number; [key: string]: any }>;
};
// Debug: save response if BIRD_DEBUG_JSON is set
if (process.env.BIRD_DEBUG_JSON) {
const fs = await import('node:fs/promises');
const debugPath = process.env.BIRD_DEBUG_JSON.replace('.json', '-explorepage.json');
await fs.writeFile(debugPath, JSON.stringify(data, null, 2)).catch(() => {});
if (debug) {
console.error(`[ExplorePage] Saved response to ${debugPath}`);
}
}
if (data.errors && data.errors.length > 0) {
return { success: false, error: data.errors.map((e) => e.message).join('; ') };
}
const items = this.parseNewsItems(data, count, aiOnly, includeRaw);
if (items.length === 0) {
return { success: false, error: 'No news items found' };
}
if (withTweets) {
await this.enrichWithTweets(items, tweetsPerItem, includeRaw);
}
return { success: true, items };
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
return { success: false, error: `Failed to fetch news: ${errorMessage}` };
} }
const data = (await response.json()) as {
// biome-ignore lint/suspicious/noExplicitAny: API response structure is complex
data?: any;
// biome-ignore lint/suspicious/noExplicitAny: API errors can have any structure
errors?: Array<{ message: string; code?: number; [key: string]: any }>;
};
// Debug: save response if BIRD_DEBUG_JSON is set
if (process.env.BIRD_DEBUG_JSON) {
const fs = await import('node:fs/promises');
const debugPath = process.env.BIRD_DEBUG_JSON.replace('.json', `-${tabName}.json`);
await fs.writeFile(debugPath, JSON.stringify(data, null, 2)).catch(() => {});
}
if (data.errors && data.errors.length > 0) {
throw new Error(data.errors.map((e) => e.message).join('; '));
}
// Parse timeline response
return this.parseTimelineTabItems(data, tabName, maxCount, aiOnly, includeRaw);
} }
// biome-ignore lint/suspicious/noExplicitAny: API response structure is complex /**
private parseNewsItems(data: any, maxCount: number, aiOnly: boolean, includeRaw: boolean): NewsItem[] { * Parse items from a GenericTimelineById response
const allItems: NewsItem[] = []; */
const seenHeadlines = new Set<string>(); private parseTimelineTabItems(
const debug = process.env.BIRD_DEBUG === '1';
if (debug) {
console.error('[ExplorePage] Processing explore_page data...');
}
if (!data.data?.explore_page) {
return [];
}
const explorePage = data.data.explore_page;
if (debug) {
console.error('[ExplorePage] Available sections:', Object.keys(explorePage.body || {}));
}
const body = explorePage.body || {};
// Check the timelines array first (For You, News tabs, etc.)
// biome-ignore lint/suspicious/noExplicitAny: exploring API structure
const timelines = (body as any).timelines || [];
if (debug) {
console.error(`[ExplorePage] Found ${timelines.length} timelines`);
}
for (const timelineObj of timelines) {
if (debug) {
console.error(`[ExplorePage] Timeline ID: ${timelineObj.id}, Label: ${timelineObj.labelText}`);
console.error(`[ExplorePage] Timeline keys:`, Object.keys(timelineObj));
}
const timeline = timelineObj.timeline;
if (timeline) {
const instructions = timeline.timeline?.instructions ?? timeline.instructions ?? [];
if (debug) {
console.error(`[ExplorePage] Timeline ${timelineObj.labelText} has ${instructions.length} instructions`);
}
const itemsFromTimeline = this.extractNewsItemsFromInstructions(
instructions,
timelineObj.__typename || 'timeline',
seenHeadlines,
maxCount,
aiOnly,
includeRaw,
);
if (debug) {
console.error(`[ExplorePage] Timeline found ${itemsFromTimeline.length} AI news items`);
}
allItems.push(...itemsFromTimeline);
if (allItems.length >= maxCount) {
break;
}
}
}
// Also check initialTimeline as fallback
if (allItems.length < maxCount && body.initialTimeline) {
const timeline = body.initialTimeline.timeline;
if (timeline) {
const instructions = timeline.timeline?.instructions ?? [];
const itemsFromInitial = this.extractNewsItemsFromInstructions(
instructions,
'initialTimeline',
seenHeadlines,
maxCount - allItems.length,
aiOnly,
includeRaw,
);
allItems.push(...itemsFromInitial);
}
}
return allItems;
}
private extractNewsItemsFromInstructions(
// biome-ignore lint/suspicious/noExplicitAny: API response structure is complex // biome-ignore lint/suspicious/noExplicitAny: API response structure is complex
instructions: any[], data: any,
source: string, source: string,
seenHeadlines: Set<string>,
maxCount: number, maxCount: number,
aiOnly: boolean, aiOnly: boolean,
includeRaw: boolean, includeRaw: boolean,
): NewsItem[] { ): NewsItem[] {
const items: NewsItem[] = []; const items: NewsItem[] = [];
const debug = process.env.BIRD_DEBUG === '1'; const seenHeadlines = new Set<string>();
// Navigate to timeline instructions
const timeline = data?.data?.timeline?.timeline;
if (!timeline) {
return [];
}
const instructions = timeline.instructions || [];
for (const instruction of instructions) { for (const instruction of instructions) {
if (instruction.type !== 'TimelineAddEntries') { if (instruction.type !== 'TimelineAddEntries') {
continue; continue;
} }
const entries = instruction.entries ?? []; const entries = instruction.entries || [];
if (debug) {
console.error(`[${source}] Processing ${entries.length} entries`);
}
for (const entry of entries) { for (const entry of entries) {
if (items.length >= maxCount) { if (items.length >= maxCount) {
break; break;
} }
if (debug) {
console.error(
`[${source}] Entry ID: ${entry.entryId}, content type: ${entry.content?.__typename || entry.content?.entryType || 'unknown'}`,
);
// Check if this is a "Today's News" or news section header
if (entry.content?.header || entry.content?.displayType === 'VerticalConversation') {
console.error(`[${source}] Found potential news section:`, entry.content?.header);
}
}
const content = entry.content; const content = entry.content;
if (!content) { if (!content) {
continue; continue;
} }
// Handle TimelineTimelineItem (single trend item) // Handle TimelineTimelineItem (single trend item)
if (content.itemContent && items.length < maxCount) { if (content.itemContent) {
if (debug && content.itemContent.is_ai_trend) {
console.error(
`[ExplorePage] Found AI trend in ${entry.entryId}:`,
JSON.stringify(content.itemContent, null, 2).substring(0, 500),
);
}
const newsItem = this.parseNewsItemFromContent( const newsItem = this.parseNewsItemFromContent(
content.itemContent, content.itemContent,
entry.entryId, entry.entryId,
@@ -290,11 +253,7 @@ export function withNews<TBase extends AbstractConstructor<TwitterClientBase>>(
} }
// Handle TimelineTimelineModule (multiple items) // Handle TimelineTimelineModule (multiple items)
const itemsArray = content?.items ?? []; const itemsArray = content?.items || [];
if (debug && itemsArray.length > 0) {
console.error(`[${source}] Module has ${itemsArray.length} items`);
}
for (const data of itemsArray) { for (const data of itemsArray) {
if (items.length >= maxCount) { if (items.length >= maxCount) {
@@ -307,19 +266,6 @@ export function withNews<TBase extends AbstractConstructor<TwitterClientBase>>(
continue; continue;
} }
if (debug) {
console.error(
`[${source}] Module item type: ${itemContent.__typename}, name: ${itemContent.name}, is_ai: ${itemContent.is_ai_trend}`,
);
}
if (debug && itemContent.is_ai_trend) {
console.error(
`[ExplorePage] Found AI trend in module ${entry.entryId}:`,
JSON.stringify(itemContent, null, 2).substring(0, 500),
);
}
const newsItem = this.parseNewsItemFromContent( const newsItem = this.parseNewsItemFromContent(
itemContent, itemContent,
entry.entryId, entry.entryId,
@@ -339,6 +285,8 @@ export function withNews<TBase extends AbstractConstructor<TwitterClientBase>>(
return items; return items;
} }
// biome-ignore lint/suspicious/noExplicitAny: API response structure is complex
private parseNewsItemFromContent( private parseNewsItemFromContent(
// biome-ignore lint/suspicious/noExplicitAny: API response structure is complex // biome-ignore lint/suspicious/noExplicitAny: API response structure is complex
itemContent: any, itemContent: any,
+115 -227
View File
@@ -23,6 +23,33 @@ const makeResponse = (overrides: Partial<ResponseLike> = {}): ResponseLike => ({
...overrides, ...overrides,
}); });
// Helper to create GenericTimelineById response structure
const makeTimelineResponse = (items: any[]) => ({
data: {
timeline: {
timeline: {
instructions: [
{
type: 'TimelineAddEntries',
entries: items.map((item, index) => ({
entryId: `test-entry-${index}`,
content: {
items: [
{
item: {
itemContent: item,
},
},
],
},
})),
},
],
},
},
},
});
describe('TwitterClient news API coverage', () => { describe('TwitterClient news API coverage', () => {
const originalFetch = global.fetch; const originalFetch = global.fetch;
@@ -32,51 +59,28 @@ describe('TwitterClient news API coverage', () => {
}); });
describe('getNews', () => { describe('getNews', () => {
it('returns news items from ExplorePage', async () => { it('returns news items from timeline tabs', async () => {
const mockFetch = vi.fn().mockResolvedValueOnce( // Mock multiple tab requests (forYou, news, sports, entertainment)
makeResponse({ const mockFetch = vi
json: async () => ({ .fn()
data: { .mockResolvedValueOnce(
explore_page: { makeResponse({
body: { json: async () =>
initialTimeline: { makeTimelineResponse([
timeline: { {
timeline: { is_ai_trend: true,
instructions: [ name: 'AI Breakthrough in Machine Learning',
{ social_context: {
type: 'TimelineAddEntries', text: 'AI · 2h ago · 15.5K posts',
entries: [ },
{ trend_url: {
entryId: 'test-entry-1', url: 'https://x.com/hashtag/AI',
content: {
items: [
{
itemContent: {
is_ai_trend: true,
name: 'AI Breakthrough in Machine Learning',
social_context: {
text: 'AI · 2h ago · 15.5K posts',
},
trend_url: {
url: 'https://x.com/hashtag/AI',
},
},
},
],
},
},
],
},
],
},
},
}, },
}, },
}, ]),
},
}), }),
}), )
); .mockResolvedValue(makeResponse({ json: async () => makeTimelineResponse([]) }));
global.fetch = mockFetch as unknown as typeof fetch; global.fetch = mockFetch as unknown as typeof fetch;
@@ -94,57 +98,24 @@ describe('TwitterClient news API coverage', () => {
}); });
it('filters to AI-only items when aiOnly is true', async () => { it('filters to AI-only items when aiOnly is true', async () => {
const mockFetch = vi.fn().mockResolvedValueOnce( const mockFetch = vi
makeResponse({ .fn()
json: async () => ({ .mockResolvedValueOnce(
data: { makeResponse({
explore_page: { json: async () =>
body: { makeTimelineResponse([
initialTimeline: { {
timeline: { is_ai_trend: true,
timeline: { name: 'AI News',
instructions: [
{
type: 'TimelineAddEntries',
entries: [
{
entryId: 'test-entry-1',
content: {
items: [
{
itemContent: {
is_ai_trend: true,
name: 'AI News',
},
},
],
},
},
{
entryId: 'test-entry-2',
content: {
items: [
{
itemContent: {
is_ai_trend: false,
name: 'Regular News',
},
},
],
},
},
],
},
],
},
},
},
}, },
}, {
}, is_ai_trend: false,
name: 'Regular News',
},
]),
}), }),
}), )
); .mockResolvedValue(makeResponse({ json: async () => makeTimelineResponse([]) }));
global.fetch = mockFetch as unknown as typeof fetch; global.fetch = mockFetch as unknown as typeof fetch;
@@ -157,9 +128,10 @@ describe('TwitterClient news API coverage', () => {
}); });
it('returns error for non-ok responses', async () => { it('returns error for non-ok responses', async () => {
// Mock all 4 default tabs (forYou, news, sports, entertainment) to return HTTP 500
const mockFetch = vi const mockFetch = vi
.fn() .fn()
.mockResolvedValueOnce(makeResponse({ ok: false, status: 500, text: async () => 'Server error' })); .mockResolvedValue(makeResponse({ ok: false, status: 500, text: async () => 'Server error' }));
global.fetch = mockFetch as unknown as typeof fetch; global.fetch = mockFetch as unknown as typeof fetch;
@@ -167,11 +139,12 @@ describe('TwitterClient news API coverage', () => {
const result = await client.getNews(10); const result = await client.getNews(10);
expect(result.success).toBe(false); expect(result.success).toBe(false);
expect(result.error).toContain('HTTP 500'); expect(result.error).toContain('No news items found');
}); });
it('returns error when API returns errors', async () => { it('returns error when API returns errors', async () => {
const mockFetch = vi.fn().mockResolvedValueOnce( // Mock all 4 default tabs to return API errors
const mockFetch = vi.fn().mockResolvedValue(
makeResponse({ makeResponse({
json: async () => ({ json: async () => ({
errors: [{ message: 'Rate limited' }, { message: 'Too many requests' }], errors: [{ message: 'Rate limited' }, { message: 'Too many requests' }],
@@ -185,30 +158,13 @@ describe('TwitterClient news API coverage', () => {
const result = await client.getNews(10); const result = await client.getNews(10);
expect(result.success).toBe(false); expect(result.success).toBe(false);
expect(result.error).toContain('Rate limited'); expect(result.error).toContain('No news items found');
expect(result.error).toContain('Too many requests');
}); });
it('returns error when no news items found', async () => { it('returns error when no news items found', async () => {
const mockFetch = vi.fn().mockResolvedValueOnce( const mockFetch = vi
makeResponse({ .fn()
json: async () => ({ .mockResolvedValue(makeResponse({ json: async () => makeTimelineResponse([]) }));
data: {
explore_page: {
body: {
initialTimeline: {
timeline: {
timeline: {
instructions: [],
},
},
},
},
},
},
}),
}),
);
global.fetch = mockFetch as unknown as typeof fetch; global.fetch = mockFetch as unknown as typeof fetch;
@@ -219,58 +175,32 @@ describe('TwitterClient news API coverage', () => {
expect(result.error).toContain('No news items found'); expect(result.error).toContain('No news items found');
}); });
it('deduplicates headlines', async () => { it('deduplicates headlines across tabs', async () => {
const mockFetch = vi.fn().mockResolvedValueOnce( const mockFetch = vi
makeResponse({ .fn()
json: async () => ({ .mockResolvedValueOnce(
data: { makeResponse({
explore_page: { json: async () =>
body: { makeTimelineResponse([
initialTimeline: { {
timeline: { is_ai_trend: true,
timeline: { name: 'Duplicate Headline',
instructions: [
{
type: 'TimelineAddEntries',
entries: [
{
entryId: 'test-entry-1',
content: {
items: [
{
itemContent: {
is_ai_trend: true,
name: 'Duplicate Headline',
},
},
],
},
},
{
entryId: 'test-entry-2',
content: {
items: [
{
itemContent: {
is_ai_trend: true,
name: 'Duplicate Headline',
},
},
],
},
},
],
},
],
},
},
},
}, },
}, ]),
},
}), }),
}), )
); .mockResolvedValueOnce(
makeResponse({
json: async () =>
makeTimelineResponse([
{
is_ai_trend: true,
name: 'Duplicate Headline',
},
]),
}),
)
.mockResolvedValue(makeResponse({ json: async () => makeTimelineResponse([]) }));
global.fetch = mockFetch as unknown as typeof fetch; global.fetch = mockFetch as unknown as typeof fetch;
@@ -283,70 +213,28 @@ describe('TwitterClient news API coverage', () => {
}); });
it('respects count parameter', async () => { it('respects count parameter', async () => {
const mockFetch = vi.fn().mockResolvedValueOnce( const mockFetch = vi
makeResponse({ .fn()
json: async () => ({ .mockResolvedValueOnce(
data: { makeResponse({
explore_page: { json: async () =>
body: { makeTimelineResponse([
initialTimeline: { {
timeline: { is_ai_trend: true,
timeline: { name: 'News 1',
instructions: [
{
type: 'TimelineAddEntries',
entries: [
{
entryId: 'test-entry-1',
content: {
items: [
{
itemContent: {
is_ai_trend: true,
name: 'News 1',
},
},
],
},
},
{
entryId: 'test-entry-2',
content: {
items: [
{
itemContent: {
is_ai_trend: true,
name: 'News 2',
},
},
],
},
},
{
entryId: 'test-entry-3',
content: {
items: [
{
itemContent: {
is_ai_trend: true,
name: 'News 3',
},
},
],
},
},
],
},
],
},
},
},
}, },
}, {
}, is_ai_trend: true,
name: 'News 2',
},
{
is_ai_trend: true,
name: 'News 3',
},
]),
}), }),
}), )
); .mockResolvedValue(makeResponse({ json: async () => makeTimelineResponse([]) }));
global.fetch = mockFetch as unknown as typeof fetch; global.fetch = mockFetch as unknown as typeof fetch;