How to Pull a Twitter Feed Into Your Site Using PHP, MySQL, and the Twitter API
These days, Twitter feeds are everywhere, it seems. There are a million widgets, gadgets, and doohickeys that let you pull your Twitter posts onto your site, but the problem with all of them is that they're tough to customize. Suppose you want to be able to grab from multiple Twitter feeds? Or omit any tweets that contain URLs or certain flagged words? With those widgets, it's tough to do. That's why I figured out how to gather my Tweets on my own. I imagine a few other people out there will want to do the same. Hopefully this tutorial will save them (you) some time. Enjoy. And, leave any related suggestions/code/questions in the comments at the bottom.
- Create a MySQL table to store the tweets. The Twitter API has limits set on how often you can ping it. So, you'll need to store your tweets locally and update it periodically using a cron job. The first step in doing that is creating a place to store the data. Copy and paste the following MySQL code into phpMyAdmin (or whatever tool you use to connect to your MySQL server):
CREATE TABLE `twitter` ( `id` BIGINT UNSIGNED NOT NULL , `screen_name` VARCHAR( 255 ) NOT NULL , `time` BIGINT UNSIGNED NOT NULL , `text` VARCHAR( 255 ) NOT NULL , `hidden` CHAR( 1 ) NOT NULL , PRIMARY KEY ( `id` ) );
- Create a cron job that grabs the tweets from the Twitter API and stores it in your table. Create a PHP file (named /cron/twitter-update.php in this example) with the following code:
<?php require_once 'db-functions.inc.php' ; //custom database functions function saveTweets($screen_name) { global $link; $screen_name = dbEscape(strtolower(trim($screen_name))); if (!$screen_name) { echo "<p><strong>Error: No screen name declared.</strong></p>\n"; return false; } $row = dbGetRow("SELECT `id` FROM `twitter` WHERE `screen_name`='$screen_name' ORDER BY `id` DESC LIMIT 1"); $last_id = $row['id']; $url = "http://api.twitter.com/1/statuses/user_timeline.xml?screen_name=$screen_name" ; if ($last_id) { $url .= "&since_id=$last_id" ; } $ch = curl_init($url); curl_setopt ($ch, CURLOPT_RETURNTRANSFER, TRUE); $xml = curl_exec ($ch); curl_close ($ch); $affected = 0; $twelement = new SimpleXMLElement($xml); foreach ($twelement->status as $status) { $text = dbEscape(trim($status->text)); $time = strtotime($status->created_at); $id = $status->id; dbQuery("INSERT INTO `twitter` (`id`,`screen_name`,`time`,`text`,`hidden`) VALUES ('$id','$screen_name','$time','$text','n')"); $affected = $affected + dbAffectedRows(); } return "<p>".number_format($affected)." new tweets from $screen_name saved.</p>\n" ; } echo saveTweets('inkplant'); ?>
- Add the cron job to your crontab file. You need to let your server know how often to run this new file that you created. In the example below, it will run every 5 minutes. Go to your Unix shell and type crontab -e and then add the following line:
*/5 * * * * /usr/bin/php /cron/twitter-update.php
Note: The settings your server uses to run cron jobs can vary a lot. Check with your server admin for instructions specific to your machine if the example above doesn't work. - Pull the databased content to your web page. Place the following code within the body of your PHP page where you'd like to display the Twitter feed. Of course, you can customize here as much as you want...
<?php echo "<ul class=\"twitter_feed\">\n" ; $result = dbQuery("SELECT * FROM `twitter` WHERE `screen_name`='inkplant' AND `hidden` != 'y' ORDER BY `time` DESC LIMIT 10"); while ($row = dbGetRow($result)) { $text = stripslashes($row['text']); $time = $row['time']; echo " <li><span class=\"twitter_time\">".date('M j, Y, g:i a',$time)."</span> · $text</li>\n" ; } echo "</ul>\n" ; ?>
This post was published on Sunday, August 22nd, 2010 by Robert James Reese in the following categories: MySQL, PHP, Twitter. Before using any of the code or other content in this post, you must read and agree to our Terms & Conditions.
73 Comments
echo saveTweets('twitterfeed1');
echo saveTweets('twitterfeed2');
But, if you decide to make it automatic, I'd recommend just adding another MySQL query at the end of the saveTweets() function that deletes the previous tweets. That way, you make sure that you get the new ones before deleting the old.
$save = array();
$result = dbQuery("SELECT `id` FROM `twitter` WHERE `hidden` != 'y' ORDER BY `time` DESC LIMIT 20");
while ($row = dbGetRow($result)) {
$save[] = $row['id'];
}
dbQuery("DELETE FROM `twitter` WHERE `id` NOT IN ('".implode('','',$save)."')");
Double check that before you use it. I just wrote it and haven't tested it.
$save = array();
$result = dbQuery("SELECT `id` FROM `twitter` WHERE `screen_name`='$screenname' ORDER BY `time` DESC LIMIT 20");
while ($row = dbGetRow($result))
{
$save[] = $row['id'];
}
$savelist = implode(",", $save);
dbQuery("DELETE FROM `twitter` WHERE `screen_name`='$screenname' AND `id` NOT IN ('$savelist')");
*this one only saves the one latest tweet so it's good for periodic cleaning out of the db
Thanks for all your help!
I am getting this error.
Call to undefined function dbconnect()
I have tried a few different things but nothing seems to be working.
Thanks
Thanks in advance.
require_once './db-functions.inc.php';
And make sure that line is at the top of the code.
Thanks again in advance.
Everything work great now.
Thanks for all your help.
Thanks very much!
echo saveTweets('teamcoco');
Only change I made was to run the output through a few regular expressions to turn hastags, urls and @usernames into hyperlinks:
$text = preg_replace('/(https{0,1}://[w-./#?&=]*)/', '$1', $text);
$text = preg_replace('/@(w+)/', '@$1', $text);
$text = preg_replace('/s#(w+)/', ' #$1', $text);
Has this something to do with the way of authing?
Right now I am using wget and running a cron job. But it is for one twitter feed search.
I am using this :
wget -O /home/myusername/public_html/yahooapi/wget-twitter.xml -q -t 1 http://search.twitter.com/search.atom?q=LSU
But I don't want to do a seperate cronjob for each search it would kill my shared hosting.
So it would be great if I can do many searches for 1 cron job.
Do u think this is possible?
Thanks
Thanks!
$screen_name = trim(strip_tags($_GET['screen_name']));
echo saveTweets($screen_name);
Any idea? Do they need to be escaped or anything?
Great tutorial! I'm having a problem getting this running though.
Here are the errors that I'm getting:
Warning: SimpleXMLElement::__construct() [simplexmlelement.–construct]: Entity: line 85: parser error : Entity 'copy' not defined in /home/riktig/public_html/lanternclub.com/twitter_update.php on line 22
Warning: SimpleXMLElement::__construct() [simplexmlelement.–construct]: © 2011 Twitter in /home/riktig/public_html/lanternclub.com/twitter_update.php on line 22
Warning: SimpleXMLElement::__construct() [simplexmlelement.–construct]: ^ in /home/riktig/public_html/lanternclub.com/twitter_update.php on line 22
It seems to be having an issue with the SimpleXMLElement but I can't seem to figure this issue out.
Thanks for this - it was very useful.
Upon reading this I was wondering, is there a way of pulling all publicly available Tweets with a certain hashtag without using the Twitter API?
http://www.inkplant.com/code/php-mysql-database-functions.php
I was wondering how did Ryan Kelley get the hastags and mentions and links to turn into hyperlinks on the output?
I would like to do that too, I'm fairly new to PHP yet and I'd like to know where to put that code to do that.
$text = preg_replace('/(https{0,1}://[w-./#?&=]*)/', '$1', $text);
$text = preg_replace('/@(w+)/', '@$1', $text);
$text = preg_replace('/s#(w+)/', ' #$1', $text);
Thanks, hope you can help.
If anyone's wondering run the output $text through expressions after
$text = stripslashes($row['text']);
and you should be good.
I ended up not using the pulldata to page content part of the script and used a much simpler one.
Thanks again for the twitter grabbing script.
You rock with the code.
I'm having a right difficulty with the preg_replace to link the hashtags and links:(
My code looks like this:
$text = stripslashes($row['text']);
$time = $row['time'];
$text = preg_replace('/(https{0,1}://[w-./#?&=]*)/', '$1', $text);
$text = preg_replace('/@(w+)/', '@$1', $text);
$text = preg_replace('/s#(w+)/', ' #$1', $text);
echo " ".date('M j, Y, g:i a',$time)." $textn" ;
but i get the error:
Warning: preg_replace() [function.preg-replace]: Unknown modifier '/'...
I've tried escaping some of the slashes but I've got no idea what im doing with regex! Any help gratefully received!
$text = 'This is an example tweet by @cowboyhazel about #regex. For more, visit http://www.inkplant.com/code/';
$time = time();
$text = preg_replace('/(https{0,1}:\/\/[A-Za-z0-9.\/#?&=+-]*)/', '<a href="$1">$1</a>', $text);
$text = preg_replace('/@([A-Za-z0-9-]+)/', '<a href="https://twitter.com/#!/$1">@$1</a>', $text);
$text = preg_replace('/#([A-Za-z0-9-]+)/', '<a href="https://twitter.com/#!/search/%23$1">#$1</a>', $text);
echo date('M j, Y, g:i a',$time).' - '.$text ;
Warning: preg_replace() [function.preg-replace]: Unknown modifier '/' in…
Do you take on freelance work?
And, no, sorry, but I'm not currently taking on any new work. I'm swamped as it is.
Sending you good vibes! sorry for posting twice, think it's something to do with the refreshing.
I have a minor issue when i run the twitter-update.php i get the following erors:
adelgado@mailer:~/public_html/twt# php twitter-update.php
PHP Notice: Undefined index: db_errors in /home/adelgado/public_html/twt/db-functions.inc.php on line 14
PHP Fatal error: Call to undefined function mysql_connect() in /home/adelgado/public_html/twt/db-functions.inc.php on line 24
I do have the db-functions.inc.php included int the twitter-update.php: e.g:
require_once './db-functions.inc.php' ; //custom database functions
any Ideas?
apt-get install php5-curl
apt-get install php5-mysql
I'm thinking it would be good to post tweets but not include reply conversations.
if (substr($text,0,1) != '@') {
I still want them to go in, just not pull out…
Not sure about "multiple calls" to get more than the last 20 tweets. The documentation says you should be able to add &count= to the end of the API call, so you get something like this:
user_timeline.xml?screen_name=$screen_name &count=200" ;
But that doesn't seem to be working.
Good karma to you
Great guide,
i was wondering about how i can alter the keyword that there is searched for?
I cannot find where to alter the screen-name.
Could you please reply to my email?
Thank you! :)
Hi Robert, i found that you use the last "echo" line to alter the screen-name. Do you know how i would need to alter the code so that i could use an $screen_name to alter what value you search for? Thank you!
You would just add an extra call to the saveTweets function in your cron. For example:
echo saveTweets('twitterfeed1');
echo saveTweets('twitterfeed2');
$retweet_count = $status->retweet_count;
Which works fine to store the value the first time the query is run, but it needs to update when the query is run again. ON DUPLICATE KEY UPDATE doesn't seem to work (I think because of "&since_id=$last_id" meaning it doesn't check the older id).
Would an if…else work, or some other action before the if statement?
Great script and it's amazing to see you posted this in 2010 and you still get so many questions!
I am utilizing your script and it's working great. I also augmented so that all my hashtags, usernames and links are hyperlinked.
Ultimately though what I am looking to do is to query numerous users…so that I populate my db based on any number of specific users I want to pull from Twitter.
I scanned through your comments above and I saw where you said to replace the hashtag function call with:
$screen_name = trim(strip_tags($_GET['screen_name']));
echo saveTweets($screen_name);
But is it possible to pull many screen names without seriously augmenting the script?
Thanks for your time.
echo saveTweets('user1');
echo saveTweets('user2');
Thanks, I figured out that I needed to build a loop, works perfect now.
One last question…at the end of the tweets I see a - then the persons username. Is that what Twitter is sending or is that your script? It does not display on twitter like that.
Thanks so much for this script, Really is great!
This is a different approach to the same thing. I am using prepared statements with PDO (recommended nowadays) and also creating an XML at the end of my script.It will output what has been retrieved. This is a work around if twitter is not always responding to your request.
http://pastebin.com/ENsYwz8L
let me know what you guys think!
First time I have seen this thing explained clearly and furthermore, the first time i have been able to tweak the code myself to get results!
dbQuery("INSERT INTO `twitter` (`id`,`screen_name`,`time`,`text`,`hidden`,`profile_image_url`) VALUES ('$id','$screen_name','$time','$text','n','$profile_image_url')");
I use it to analyze on of the most popular radio stations in Austria as I have the impression that the play the same songs up to 10 times a day …
I need to pull #tag feeds from twitter with pics of each user who twitted with a specific hashtag. Also be able to edit modify and or delete. These data will be later posted
Leave a Comment