What is a regular expression? We can talk about that in detail some other time. Briefly, it is a string of symbols that is designed to match a specified set of symbols, or a range of a set of symbols, in a larger body or stream of text. For example, if you pass a stream of information (say, all your emails) through a filter with the regular expression:
‘\d\d\d-\d\d\d-\d\d\d\d’
then any part of that stream of information that looks like a phone number (not using parens), such as 636-555-3226, will be isolated.
Automate the Boring Stuff with Python is a book that teaches beginning Python computer Augean programming focusing on examples from day to day life, including but well beyond REs.
The new edition includes pattern matching with regular expressions, input validation, reading and writing files, organizing files, web scraping, manipulating Excel spreadsheets and Google Sheets, PDF and Word documents, CSV and JSON files, email, images, and automating your keyboard and mouse.
The great benefit of a book like this is that you learn Python (the first part of the book gives you all you need to know to program in Python) in the context of things you actually want to do with Python. If you are interested in learning Python, or coding in general, this can be your first book.
The book is well done, as all in this series are, and fun. There are strong on line resources including all the code, and that information is regularly updated. Generally, “No Starch” press books are great, and this is one of those!
I would like to have seen at least sidebars on manipulating things using Libreoffice software, but note that the book focuses on documents, and OpenSource software does work with normal Excel and Word documents, so it is there.
The second edition adds a new chapter on input validation. The Gmail and Google Sheets sections, and the information on CSV files is also new. I plan on using the software tips and tricks to develop my own highly specialized and targeted search software. I’m often looking for files that have specific extensions, and certain kinds of content, in certain locations. Just the ability to hard-wire where to search for files will save me a lot of time and trouble.
Author Al Sweigart is a professional software developer who teaches programming to kids and adults, and who is author of Invent Your Own Computer Games with Python, Cracking Codes with Python, and Coding with Minecraft, all of which are quite nice. We need a new edition of Coding with Minecraft, by the way, that looks at a wider range of coding options and keeps up with the major advances in that software environment! So, get to work, Al!
OpenSource science means, among other things, using OpenSource software to do the science. For some aspects of software this is not important. It does not matter too much if a science lab uses Microsoft Word or if they use LibreOffice Write.
However, since it does matter if you use LibreOffice Calc as your spreadsheet, as long as you are eschewing proprietary spreadsheets, you might as well use the OpenSource office package LibreOffice or equivalent, and then use the OpenSource presentation software, word processor, and spreadsheet.
OpenSource programs like Calc, R (a stats package), and OpenSource friendly software development tools like Python and the GPL C Compilers, etc. do matter. Why? Because your science involves calculating things, and software is a magic calculating box. You might be doing actual calculations, or production of graphics, or management of data, or whatever. All of the software that does this stuff is on the surface a black box, and just using it does not give you access to what is happening under the hood.
But, if you use OpenSoucre software, you have both direct and indirect access to the actual technologies that are key to your science project. You can see exactly how the numbers are calculated or the graphic created, if you want to. It might not be easy, but at least you don’t have to worry about the first hurdle in looking under the hood that happens with commercial software: they won’t let you do it.
Direct access to the inner workings of the software you use comes in the form of actually getting involved in the software development and maintenance. For most people, this is not something you are going to do in your scientific endeavor, but you could get involved with some help from a friend or colleague. For example, if you are at a University, there is a good chance that somewhere in your university system there is a computer department that has an involvement in OpenSource software development. See what they are up to, find out what they know about the software you are using. Who knows, maybe you can get a special feature included in your favorite graphics package by helping your new found computer friends cop an internal University grant! You might be surprised as to what is out there, as well as what is in there.
In any event, it is explicitly easy to get involved in OpenSource software projects because they are designed that way. Or, usually are and always should be.
The indirect benefit comes from the simple fact that these projects are OpenSource. Let me give you an example form the non scientific world. (it is a made up example, but it could reflect reality and is highly instructive.)
Say there is an operating system or major piece of software competing in a field of other similar products. Say there is a widely used benchmark standard that compares the applications and ranks them. Some of the different products load up faster than others, and use less RAM. That leaves both time (for you) and RAM (for other applications) that you might value a great deal. All else being equal, pick the software that loads faster in less space, right?
Now imagine a group of trollish deviants meeting in a smoky back room of the evile corporation that makes one of these products. They have discovered that if they leave a dozen key features that all the competitors use out of the loading process, so they load later, they can get a better benchmark. Without those standard components running, the software will load fast and be relatively small. It happens to be the case, however, that once all the features are loaded, this particular product is the slowest of them all, and takes up the most RAM. Also, the process of holding back functionality until it is needed is annoying to the user and sometimes causes memory conflicts, causing crashes.
In one version of this scenario, the concept of selling more of the product by using this performance tilting trick is considered a good idea, and someone might even get a promotion for thinking of it. That would be something that could potentially happen in the world of proprietary software.
In a different version of this scenario the idea gets about as far as the water cooler before it is taken down by a heavy tape dispenser to the head and kicked to death. That would be what would certainly happen in the OpenSource world.
So, go OpenSource! And, read the paper from Linux Journal, which by the way has been producing some great articles lately, on this topic.
The Scientists Workflow and Software
You collect and manage data. You write code to process or analyze data. You use statistical tools to turn data into analytically meaningful numbers. You make graphs and charts. You write stuff and integrate the writing with the pretty pictures, and produce a final product.
The first thing you need to understand if you are developing or enhancing the computer side of your scientific endevour is that you need the basic GNU tools and command line access that comes automatically if you use Linux. You can get the same stuff with a few extra steps if you use Windows. The Apple Mac system is in between with the command line tools already built in, but not quite as in your face available.
You may need to have an understanding of Regular Expressions, and how to use them on the command line (using sed or awk, perhaps) and in programming, perhaps in python.
You will likely want to master the R environment because a) it is cool and powerful and b) a lot of your colleagues use R so you will want to have enough under your belt to share code and data now and then. You will likely want to master Python, which is becoming the default scientific programming language. It is probably true that anything you can do in R you can do in Python using the available tools, but it is also true that the most basic statistical stuff you might be doing is easier in R than Python since R is set up for it. The two systems are relatively easy to use and very powerful, so there is no reason to not have both in your toolbox. If you don’t chose the Python route, you may want to supplement R with gnu plotting tools.
You will need some sort of relational database setup in your lab, some kind of OpenSource SQL lanaguge based system.
You will have to decide on your own if you are into LaTex. If you have no idea what I’m talking about, don’t worry, you don’t need to know. If you do know what I’m talking about, you probably have the need to typeset math inside your publications.
Finally, and of utmost importance, you should be willing to spend the upfront effort making your scientific work flow into scripts. Say you have a machine (or a place on the internet or an email stream if you are working collaboratively) where some raw data spits out. These data need some preliminary messing around with to discard what you don’t want, convert numbers to a proper form, etc. etc. Then, this fixed-up data goes through a series of analyses, possibly several parallel streams of analysis, to produce a set of statistical outputs, tables, graphics, or a new highly transformed data set you send on to someone else.
If this is something you do on a regular basis, and it likely is because your lab or field project is set up to get certain data certain ways, then do certain things to it, then ideally you would set up a script, likely in bash but calling gnu tools like sed or awk, or running Python programs or R programs, and making various intermediate files and final products and stuff. You will want to bother with making the first run of these operations take three times longer to set up, so that all the subsequent runs take one one hundredth of the time to carry out, or can be run unattended.
Nothing, of course, is so simple as I just suggested … you will be changing the scripts and Python programs (and LaTeX specs) frequently, perhaps. Or you might have one big giant complex operation that you only need to run once, but you KNOW it is going to screw up somehow … a value that is entered incorrectly or whatever … so the entire thing you need to do once is actually something you have to do 18 times. So make the whole process a script.
Aside form convenience and efficiency, a script does something else that is vitally important. It documents the process, both for you and others. This alone is probably more important than the convenience part of scripting your science, in many cases.
Being small in a world of largeness
Here is a piece of advice you wont get from anyone else. As you develop your computer working environment, the set of software tools and stuff that you use to run R or Python and all that, you will run into opportunities to install some pretty fancy and sophisticated developments systems that have many cool bells and whistles, but that are really designed for team development of large software projects, and continual maintenance over time of versions of that software as it evolves as a distributed project.
Don’t do that unless you need to. Scientific computing often not that complex or team oriented. Sure, you are working with a team, but probably not a team of a dozen people working on the same set of Python programs. Chances are, much of the code you write is going to be tweaked to be what you need it to be then never change. There are no marketing gurus coming along and asking you to make a different menu system to attract millennials. You are not competing with other products in a market of any sort. You will change your software when your machine breaks and you get a new one, and the new one produces output in a more convenient style than the old one. Or whatever.
In other words, if you are running an enterprise level operation, look into systems like Anaconda. If you are a handful of scientists making and controlling your own workflow, stick with the simple scripts and avoid the snake. The setup and maintenance of an enterprise level system for using R and Python is probably more work before you get your first t-test or histogram than it is worth. This is especially true if you are more or less working on your own.
Culture
Another piece of advice. Some software decisions are based on deeply rooted cultural norms or fetishes that make no sense. I’m an emacs user. This is the most annoying, but also, most powerful, of all text editors. Here is an example of what is annoying about emac. In the late 70s, computer keyboards had a “meta” key (it was actually called that) which is now the alt key. Emacs made use of the metakey. No person has seen or used a metakey since about 1979, but emacs refuses to change its documentation to use the word “alt” for this key. Rather, the documentation says somethin like “here, use the meta key, which on some keyboards is the alt key.” That is a cultural fetish.
Using LaTeX might be a fetish as well. Obliviously. It is possible that for some people, using R is a fetish and they should rethink and switch to using Python for what they are doing. The most dangerous fetish, of course, is using proprietary scientific software because you think only if you pay hundreds of dollars a year to use SPSS or BMD for stats, as opposed to zero dollars a year for R, will your numbers be acceptable. In fact, the reverse is true. Only with an OpenSource stats package can you really be sure how the stats or other values are calculated.
This book focuses on Python and not R, and covers Latex which, frankly, will not be useful for many. This also means that the regular expression work in the book is not as useful for all applications, as might be the case with a volume like Mastering Regular Expressions. But overall, this volume does a great job of mapping out the landscape of scripting-oriented scientific computing, using excellent examples from biology.
Mastering Regular Expressions can and should be used as a textbook for an advanced high school level course to prep young and upcoming investigators for when they go off and apprentice in labs at the start of their career. It can be used as a textbook in a short seminar in any advanced program to get everyone in a lab on the same page. I suppose it would be treat if Princeton came out with a version for math and physical sciences, or geosciences, but really, this volume can be generalized beyond biology.
Stefano Allesina is a professor in the Department of Ecology and Evolution at the University of Chicago and a deputy editor of PLoS Computational Biology. Madlen Wilmes is a data scientist and web developer.
It covers how Python works, so someone familiar with programming languages can get up to speed. Then, the book tackles a number of key important tasks one may use a computer for. This includes working with Regular Expressions, file reading and writing, web scraping, interacting with Excel spreadsheets and PDF files, scheduling things, working with email, manipulating images, and messing around with the keyboard and mouse.
I wold like to see a second volume with yet more programming ideas and examples. It could be a series.
From the publishers:
If you’ve ever spent hours renaming files or updating hundreds of spreadsheet cells, you know how tedious tasks like these can be. But what if you could have your computer do them for you?
In Automate the Boring Stuff with Python, you’ll learn how to use Python to write programs that do in minutes what would take you hours to do by hand—no prior programming experience required. Once you’ve mastered the basics of programming, you’ll create Python programs that effortlessly perform useful and impressive feats of automation to:
Search for text in a file or across multiple files
Create, update, move, and rename files and folders
Search the Web and download online content
Update and format data in Excel spreadsheets of any size
Split, merge, watermark, and encrypt PDFs
Send reminder emails and text notifications
Fill out online forms
Step-by-step instructions walk you through each program, and practice projects at the end of each chapter challenge you to improve those programs and use your newfound skills to automate similar tasks.
In the first half of the book, you’ll learn about basic programming concepts, such as lists, dictionaries, classes, and loops, and practice writing clean and readable code with exercises for each topic. You’ll also learn how to make your programs interactive and how to test your code safely before adding it to a project. In the second half of the book, you’ll put your new knowledge into practice with three substantial projects: a Space Invaders-inspired arcade game, data visualizations with Python’s super-handy libraries, and a simple web app you can deploy online.
Scratch, the colorful drag-and-drop programming language, is used by millions of first-time learners, and in Scratch Programming Playground, you’ll learn to program by making cool games. Get ready to destroy asteroids, shoot hoops, and slice and dice fruit!
Each game includes easy-to-follow instructions, review questions, and creative coding challenges to make the game your own. Want to add more levels or a cheat code? No problem, just write some code.
Invent Your Own Computer Games with Python will teach you how to make computer games using the popular Python programming language–even if you’ve never programmed before!
Begin by building classic games like Hangman, Guess the Number, and Tic-Tac-Toe, and then work your way up to more advanced games, like a text-based treasure hunting game and an animated collision-dodging game with sound effects. Along the way, you’ll learn key programming and math concepts that will help you take your game programming to the next level.
Want to introduce kids to coding in a fun and creative way?
With the Scratch Coding Cards, kids learn to code as they create interactive games, stories, music, and animations. The short-and-simple activities provide an inviting entry point into Scratch, the graphical programming language used by millions of kids around the world.
Kids can use this colorful 75-card deck to create a variety of interactive programming projects. They’ll create their own version of Pong, Write an Interactive Story, Create a Virtual Pet, Play Hide and Seek, and more!
Each card features step-by-step instructions for beginners to start coding with Scratch. The front of the card shows an activity kids can do with Scratch–like animating a character or keeping score in a game. The back shows how to put together code blocks to make the projects come to life! Along the way, kids learn key coding concepts, such as sequencing, conditionals, and variables.
This collection of coding activity cards is perfect for sharing among small groups in homes and schools.
To cut to the chase, there are two reasons I recommend this book. First, the specific programming projects used in the book are a good match for most people, because they are bare bones (but highly developed) exemplars that are fairly adaptable and together cover a wide range of applications and use requirements. Second, the book is well written and organized, the first part very serviceable as a reference book, covers both Python 2 and 3 but focuses on and encourages you to learn 3 (which you should) etc.
Let me go back to that first reason and expand a bit.
Why do you want to learn Python, why do you want to program stuff?
You need to automate or otherwise develop an interactive project. You need to manipulate data live, interact with the computer, etc. You have some data in a text file (or some other form) and you need to access it and turn it into derived numbers, or pictures, graphics, etc. You want to generate web output. Perhaps there is some web scraping in there. Maybe you are doing all this together.
The book begins with eleven chapters on how Python works, and is fairly detailed. If you work through this in detail, and actgually do the recommended exercises, you’ll be a python programmer before you hit Chapter 11.
The second part includes three fairly well developed projects. One is an interactive game that is pretty sophisticated (for a scripting language an not using a GUI). The second uses some of the more powerful mathematical and graphical libraries in Python to manipulate, graph, plot, etc. data. This section also covers working with API’s including Git. This is probably the chapter you’ll come back to the most.
The third project leads you through developing a web application using Django.
A classic use of this book is that you are a scientists who uses R (r-cran) and you suddenly realize that more development of tools is happening in Python than in R. Switching from R to Python is hard to do emotionally, but easy from a programming perspective, because Python is a better programming language. You don’t really want to leave R, but you know that it is time to branch out, and at least, see what you can do with Python. This crash course does not give you the full range of knowledge to switch you from sophisticated use of R to equivalent use of Python, but if you can’t currently program in Python, do this, then do that using more sophisticated resources.
It has been interesting to see, over the last few years, No Starch Press, which produces this book, growing and producing future classics that should be along side the more traditional O’Reilly Press programming books. Python Crash Course: A Hands-On, Project-Based Introduction to Programming is one of those books, equal to or replacing something like Learning Python, 5th Edition, but at close to half the price.
Python is easy to use and learn, yet it is also very powerful. Much of the power comes from the powerful libraries that exist, which can be imported and used for a wide range of things. Python itself is a very simple implementation of an interpreted language, with a simple command line interface. Because of these two things, the actual installation and running of Python is very easy and sometimes very difficult at the same time. Here’s the thing. As a single user who may do some complex stuff, which would describe you if you are like me, you might want a pretty fancy development environment and lots of libraries and stuff. But at the same time, you really don’t want complex virtual environments and collaboration tools. The thing is, as the various free or paid add ons or resources you can get to enhance Python’s power get more complicated, they assume that you are moving from a hobbyist or student to a corporate environment with multiple collaborators and the need to keep projects separate more than you really want. At some point, someone will tell you, “Oh, if you want to do that, just install ______” where the blank is the name of a snake or something. You go in stall it, and find out you have to take a class to know what the first button to press is.
So, that is a complaint I have about the Python world. This book does come with a web site that has on it current and important information, updated, on how to handle some of these problems with installing and configuring your programming environment, using a thing called “pip” which helps you install libraries and stuff, and how to get matplotlib and some other stuff running without having to take that class.
You will also find source code used in the book and some other cool resources on that page.
Chapter 1: Getting Started
Chapter 2: Variables and Simple Data Types
Chapter 3: Introducing Lists
Chapter 4: Working with Lists
Chapter 5: if Statements
Chapter 6: Dictionaries
Chapter 7: User Input and while Loops
Chapter 8: Functions
Chapter 9: Classes
Chapter 10: Files and Exceptions
Chapter 11: Testing Your Code
PART II: Projects
Project 1: Alien Invasion
Chapter 12: A Ship that Fires Bullets
Chapter 13: Aliens!
Chapter 14: Scoring
Project 2: Data Visualization
Chapter 15: Generating Data
Chapter 16: Downloading Data
Chapter 17: Working with APIs
Project 3: Web Applications
Chapter 18: Getting Started with Django
Chapter 19: User Accounts
Chapter 20: Styling and Deploying an App
Afterword
Appendix A: Installing Python
Appendix B: Text Editors
Appendix C: Getting Help
Appendix D: Using Git for Version Control
View the detailed Table of Contents (PDF)
View the Index (PDF)
There is really no better time to get a Raspberry Pi. The new Raspberry Pi 3 has features that make it much more useful and fun, including more speed, built in bluetooth, and built in wifi.
The Raspberry Pi is a small computer that, out of the box, lacks storage drive or device, a monitor, a screen, or a mouse, but is otherwise a fully functional computer that can run a normal operating system. It costs very little, so if you happen to have a TV or monitor that can use a component or HDMI hookup, a keyboard, a mouse, and an appropriate microSD card, then you have a computer for $39.99. If, on the other hand, you need to buy those parts and the cables and such, then you have a $200 computer (or more) that isn’t very fast but is still cool. Chances are, though, that you have at least some of these parts laying around.
I’ll make some suggestions below as to how to set yourself up (including the option of buying a kit) but first let me say a little more about why you might want to do this.
Why Pi reason one: A kid
Maybe you have a kid in the house who is old enough to start to play around with a computer, but not old enough to not gob all over it, drop it, or mistake it for a bath toy. With a Raspberry Pi, you are not taking a big risk. First, it is cheap and easily replaced, compared to, say, a laptop. Second, if it does end up in the bath, it is probably easier to dry out than a notebook or desktop computer.
When Huxley was about five, I set him up with a Raspberry Pi hooked to an old TV we had laying around. I bought a new keyboard and mouse for it, and that keyboard and mouse has done a lot of good work since. He learned the basics of turning a computer on, turning a computer off, using the mouse, running some basic apps (like Tux Paint), etc.
Over the next year he continued to use the Pi but also graduated to a laptop his grandfather had laying around. It was a Windows laptop, so I wiped it and installed Linux. Huxley is now able to install software using the command line, and he is about as good at word processing as any average person. He’s pretty good with graphics software as well. At school, he’s been learning a watered down version of the iMac, he can use Windows, and he knows more about the Android and iOS operating systems than most. He turned 7 last week. I owe much of his rapid development to the Raspberry Pi. Also, to Linux; If Linux is one of your first operating systems, you will understand more quickly the things you need to know to make computers bend to your will.
Why Pi reason two: impress your friends
So, here’s the scenario. A bunch of your friends get together now and then at a particular coffee shop, or the cafeteria, or perhaps your family gathers at the cabin, or at a favorite vacation spot. Pretty soon everyone pulls out a device. Most have phones, but here and there a laptop being used by someone who needs to get some real work done.
Well, you need to get some real work done too, so you pull out a box of Altoids, plug it into the nearest TV, sign in via your iPad and, using the iPad’s virtual keyboard, get to work.
A Raspberry Pi can be cobbled together with a number of different bits and pieces, including some interesting options for the computer’s case, to make what is essentially a laptop or a funky desktop. It can work as a “headless” server, so you use a different device to access it.
My most elaborate setup involved a super fast USB stick to run the operating system. The microSD is the usual boot device, but I hacked it to hand the boot process over to the faster USB stick and run the computer from there. An HDMI output ran to a small TV. The Pi was hooked directly into the internet via a LAN cord (there is a plug on the Pi). A powered USB hub allowed multiple additional possibilities. A bluetooth dongle gave the device that capability, and allowed use of a mouse and keyboard. Oh, and I overclocked it. That was a Raspberry Pi 2.
The Pi 3 comes with built in wireless and bluetooth, and is faster and thus not so much in need of overclocking.
Why Pi reasons three through infinity: Projects
The Raspberry Pi is, as noted, a basic computer, but it has a number of pins that give the hardware access to outside thingies. These GPIO pins let you hook sensors or actuators directly to the Pi, so you can build things with a brain. In addition, there is the usual serial (USB) interface.
There are two commonly used machines on which a very large number of Do It Yourself (DIY) projects have been based. A home media center. A garage door opener. A weather station. A personal web server. A remote control camera for security or to watch birds or game. A system for turning on and off all your things, including lights. A customized alarm clock. A bitcoin server. Any one of a number of different robots.
One device is the Arduino Uno R3 Microcontroller, or a related card, the other is the Raspberry Pi. If you look around you will find dozens of other circuit boards that do these things, but none of them have the huge support network, gazillion lines of pre-written code, and myriad bits of hardware designed to be added on to make things happen.
The Arduino controller is, frankly, the better choice of machine for many uses. If you need a smartish machine to do one thing, and only one thing, and that thing isn’t too complicated, a Raspberry Pi is probably overkill and might have some downsides. A controller might even be better designed as hardware goes, as it is real time. Say you are building a model airplane and you want a smart device inside it controlling the various things that make it fly. You will be sending signals via radio to the plane. A controller can be set to cycle rapidly through the communications module to see if there are any instructions coming in, then to various sensor or actuator circuits to acquire information or to set the actuators to do things (change speed, etc.). You can be confident in advance that the controller will focus on nothing other than these jobs, and check each of these inputs and outputs several times (a known number of times) a second.
If, on the other hand, you use a computer — any typical computer — with a normal operating system like Linux, OSX, or Windows, then you can’t be as sure. There can be a software based timer set up in the computer that will theoretically go from sensor or actuator to sensor or actuator, checking the communication module for commands every cycle. But those cycles are virtual. If somewhere along the way a backup program was installed to make a copy of a directory and put it on a USB stick every hour, then suddenly your computer is busy doing something else. If updates are automatic on your setup, same problem. Yes, you can configure a system to not do anything you don’t want it to do, but you might not be perfect in you efforts. In other words, having a Raspberry Pi carry out time sensitive mission critical rapid fire acts is like hiring a potentially recidivistic but reformed fox to guard the henhouse. Yeah, it probably works, but if any chickens are missing …
The Raspberry Pi, running a normal operating system, can do pretty much anything any other computer can do, and might in fact, do any of a number of different things while still carrying out the program you wrote to operate the air plane. But the Arduino controller can only do a few things, and you can’t really accidentally have it programmed to send out bulk emails or get your fur coat cleaned while you are trying to control your expensive model airplane.
So, let’s say I want a device in my garage that has a sensor that tells me if the garage door is open or closed, turning on a light on a second device in my house (connected by radio signal) if the door is open, and allowing me to open or close the garage door by pushing a button on that interior device. That would be easy using two Arduino boards, some sort of sensor, a cheap pair of radio communication thingies, and some lights and resistors. (Plus the appropriate way to close/open the door, for which there are several options.)
Say I want to have a device in the garage that senses if the door is open, has a camera that can be used to check out the garage, communicates the status of the door via the internet, and allows me to remotely open or close the door or look at the camera’s image remotely form anywhere on my smart phone. You can totally do that with an Arduino controller, but that will involve adding a lot of parts and raising the cost, and you may be limited in other ways. A Raspberry Pi would be better for this application.
Say I want the device in the garage to have a movable pointable camera, a temperature sensor, a “door open” sensor, a door close/open actuator, and a way to turn on and off the main garage light. But, I want to control all of those functions from the Internet, and I want the same internet interface to also give me the temperature in my house and provide a view from a camera pointing out the front door. I also want the same system to give me access to my Google calendar.
For that, I’d want the Raspberry Pi running the web server, hard wired, perhaps, into my home LAN, and fitted with a Camera Module and a temperature sensor. I the Garage would be an Arduino with a movable camera and all that other stuff, controlled via the Pi that sits in my house.
And so on.
Very elaborate home automation systems have been built using a combination of various Arduino controllers and a Raspberry Pi at the center of it all.
The Raspberry Pi can run any software that normally runs on Linux, if that is the operating system you install (which is probably the case). Most projects that use the Pi as the central brain use Python, which is an all purpose programming language. The Arduino controller uses a form of C, but it is easy to use and for most functions, there is already code ready to use out of the box or with minor changes. The Pi can have huge programs and handle massive amounts of data, if you have a large enough USB stick or microSD card. The Arduino is very limited in space and memory, though it can be attached, with some extra work, to various mass storage devices.
I think you get the idea.
What kind of Pi do you want?
There are four kinds of Raspberry Pi. The Zero, the One, the Two, and the Three. If you are getting a new one, just get the three. It replaces the one and the two. (The Zero is a whole different thing, for advanced users.)
There is also the A vs. B model within these categories. None of that is important. Get the Pi 3 B. THIS ONE: Raspberry Pi 3 Model B Motherboard.
In order to make a Raspberry Pi work, you will need things like a power adapter, a keyboard, etc. You might want to put it in a case. You will need a microSD card with the operating system on it. Since the Pi is a DIY device, the basic unit comes with none of this and you have to figure it all out yourself.
Or, you can buy any one of dozens of available kits that have some combination of some of the basic parts needed. Kits don’t have a monitor, keyboard, or mouse. The Raspberry Pi comes with the rarely used component jack (the yellow round thing on the back of your TV may work with it) and an HDMI output, for video, and several USB sockets. The PI 3 has bluetooth built in, so you can use a bluetooth mouse and keyboard if you want. The kits tend to have the Pi, an SD card often with the OS preloaded, an HDMI cable, a power supply, and a case. Most kits these days come with heat sinks, which are pretty much unnecessary but cool, that you can glue on to the Pi if you want.
I recently got the Vilros Raspberry Pi 3 Complete Starter Kit with Clear Case and 32GB SD Card, which came with the Pi 3 (with WiFi and Bluetooth, as mentioned), a power supply, a 32 GB Class 10 MicroSD card (you will want a class 10 card), a case, an HDMI cord, and two heat sinks. I think that normally lists for $90 but is perpetually on sale for much less.
For the more DIY oriented user, the CanaKit Raspberry Pi 3 Ultimate Starter Kit – 32 GB Edition comes with additional parts that will be helpful in building projects that exploit the Pi’s input/output pins, such as LED lights, hookup wires, and resistors, a breakout cable, and a breadboard. The way the Pi is set up, you will want a breakout system to allow you to mess with electronic components more easily and safely (safe for the Pi, that is, with its sensitive computer circuity). If I was going to give a kit to someone who I knew was going to mess around with the hardware, this is the kit I’d probably pick.
The Operating System, which you can download at the Raspberry Pi site and install on the microSD card, also allows for some options. And, it can be a little confusing.
There is a “system” called “noobs” (for Noobie, I assume) which is the easy way to install Raspbian. Or, you can get Rraspbian sans Noob, which is apparently the hard way to install Raspbian. I’ve done both, but I don’t remember if one is actually harder.
Raspbian is the basic OS for the Raspberry Pi, and it is a Linux distribution based on Debian (get it? Raspberry Pi Debian = Raspbian?)
The Raspbian distribution formerly used LXDE as the desktop, a lightweight, not very fancy, stable desktop. But, recently, the desktop was redesigned to be much sleeker and nicer (but still basic and efficient) and has been renamed Pixel. (Pixel as in Pi-something, but also, named in part after the author’s experiences as a young child playing peek-a-poke in Basic on his IBM clone.)
You can also install a version of Ubuntu Mate. I’ve done that. I don’t recommend it. The Pi is not meaty enough to pretend to be a normal desktop computer when saddled with the extra overhead. Pixel is the way to go. But it was fun to play around with. Another option is the OSMC, or Open Source Media Center, if you want to use the machine as a media center. There are others. I’ve no experience with them, so I can’t tell you much about them. But I can tell you this very cool thing: You put the OS on the microSD, and put the microSD in the Pi, and that is what determines what OS you are using. So, you can swap between operating systems but just swapping cards. How cool is that?
One especially advanced but seemingly widespread use of the Pi is to build a gaming device. You install some software. Then, you install the bios of one or more gaming systems. Then you install the games that go with that gaming system. Obviously, you will only install systems and games that you already own, and not find anyplace on the Internet to download them. Then, you have all those old console games on your Pi. You’d probably buy some game controllers to hook up to that device, and you’d probably run it on your TV.
All the information you need to make all of this work can be found either on the Raspberry Pi site, or at a destination linked to from that site.
Raspberry Pi Cases
As noted, you can use the Raspberry pi as a regular, low power but usable, computer and at the same time make a cool case for it. This is a bit like having a laptop that you carry around, but instead of a laptop, it is a box of Altoids. Or some LEGOS.
Instructions to make the Altoids Pi case are here.
There are a gazillion projects on line that use the Raspberry Pi. Here, I’ll give you just a few examples. Most of the examples use Raspberry Pi 1 or 2 (probably, 2 is the most common) and in most cases you’ll want to use a 3. So, the best way to get good instructions for a project is to look at a few examples and hopefully among them will be one that uses the Pi 3.
Here is a somewhat slow but a good start on a PIrsonal Assistant. You can make this run faster if you tweak it (and use the Pi 3):
Do you know what Tor is? If not, maybe find out. Given the current political climate, it may become more important. Anyway, you can make a Tor Router.
There are many picture frame projects out there. Just search for Raspberry Pi Picture Frame.
Here is a system that controls your lights using voice commands.
If it was me, I’d skip the voice feedback. The detailed instructions are HERE.
At first I thought this Pi controlled espresso machine would be dumb, because really, what can a computer to do make an espresso machine work better. Then I looked more closely at it and realized this is a thing.
Stratux looks interesting. I remember my uncle, a Franciscan missionary who would stop by now and then on his way to this or that remote place, invented something like this back in the 60s. Stratus is a device you use to identify aircraft in your vicinity, using the plane’s ADS-B broadcasts.
I’m not going to point to a particular media streaming project. There are many, many such projects and it all depends on what you want to do. Just do a search for using the OSMC system on Pi. Or, just a simple music streaming machine.
Same with console gaming projects.
People make Minecraft servers. Generally, if there is anything you might like to dedicate a computer to, that is on all the time, the Pi may be the way to go. It is very low power demanding, and cheap enough that you can dedicate this card to that one purpose, leave it running, and give it just enough maintenance to keep it from being coopted by nefarious hackers.
A print server, a file server, a local web server for your own uses, a system that puts your google calendar or other stuff on a TV screen (or other screen) for your convenience. Each of these idea has 20 or 30 iterations out there.
I hear someone is working on an “Election Converter” that runs on a Raspberry Pi 3. I’d send you a link but the instructions are written in Cyrillic.
Minecraft is a gaming world. Or, if you like, a “sandbox.” This is a three dimensional world in which characters do things, all sorts of things. The context for the world of Minecraft is very open ended. The player builds things, moves things, gets things, does things, in a way that makes any one gamer’s game potentially very different from any other gamer’s game.
If you install Minecraft from Minecraft.net (about 30 bucks) and have Python 3, Java, the Minceraft Python API, and a Spigot Minecraft Server, you can program your own versions of the game using Python programming/scripting language.
But how? How do you do that?
Well, you can get Learn to Program with Minecraft: Transform Your World with the Power of Python. This book is intended to teach programming, in the Minecraft setting. The book is designed for kids 10 years and older, though I’m sure some younger kids can use it. Also, it must be admitted that a learning to program book like this may be most valuable for adults who are not coders but want to learn some coding, and happen to be gamers and like Minecraft.
The book, new on the market, provides excellent instructions for setting up all that stuff mentioned above. Everything should work on a Windows machine, on Mac OS X, and Linux.
The programming you do with this book is pretty sophisticated. You learn to create palaces, pyramids, to teleoport players around, to stack blocks, interact with Minecraft’s chat feature, blow stuff up, cast spells, and replicate sections of the Minecraft countryside.
Here is what is interesting about this approach. Python programming is pretty basic, and pretty useful, but one has to do a lot of work to develop something slick and fancy and highly functional (counting working video games or interfaces as highly functional). But working with the existing Minecraft system, via the API, allows some relatively simple programming to produce impressive results. This is “Hello World” on steroids, at the very least.
Of all the diverse No Starch Press programming guides, this one may turn out to be the most effective, as a teaching tools, for that special case where a person is already interested in Minecraft and wants to learn Python.
Here is the Table of Contents:
Introduction
Chapter 1: Setting Up for Your Adventure
Chapter 2: Teleporting with Variables
Chapter 3: Building Quickly and Traveling Far with Math
Chapter 4: Chatting with Strings
Chapter 5: Figuring Out What’s True and False with Booleans
Chapter 6: Making Mini-Games with if Statements
Chapter 7: Dance Parties and Flower Parades with while Loops
Chapter 8: Functions Give You Superpowers
Chapter 9: Hitting Things with Lists and Dictionaries
Chapter 10: Minecraft Magic with for Loops
Chapter 11: Saving and Loading Buildings with Files and Modules
Chapter 12: Getting Classy with Object-Oriented Programming
Afterword
Block ID Cheat Sheet
The author, Craig Richardson, is a teacher of Python, former high school computing science teacher, and has been involved with the Raspberry Pi Foundation.
My first computer language was PL/1, but soon after I learned, among other languages, Basic, and I really liked Basic and I still do. Basic is linear, and I think in linear constructs when I do any kind of computer program. This is probably, in part, because user interfaces are the last thing I want to deal with. I want a series of numbers to be treated in a certain way, or a set of formulas to generate a database. The most non-linear I tend to get is multidimensional arrays, and that’s still linear.
Python is potentially, and in practice, very different, and is essentially used as an object-oriented language. Yet at the same time it can be used in any other way, to reproduce pretty much any sort of programming paradigm. People thought of Basic as not very readable, but in fact, it was in its more advanced form if you programmed right. Python is said to enforce readability, if by readability we mean enforced indentation. People are still free to ruin readability in a number of other ways. But most importantly, Python holds a very important feature in common with Basic: It is interpreted. In other words, at any point in time while you are writing your Python program, you can “run” it and see how it is going.
The biggest difference between a language like Basic even at its high water mark some years ago, and Python is that Phython has plenty of modules for use do do all sorts of cool things. I’m not sure if the Python library is the biggest and vastest and most amazingest of all, but it probably is. So, if you are going to pick a programming language with paradigmatic flexibility, reasonable readability, and a powerful and diverse library of functionality, the Python is probably the way to go.
Python for Kids is a lighthearted introduction to the Python programming language, full of fun examples and color illustrations. Jason Briggs begins with the basics of how to install Python and write simple commands. In bite-sized chapters, he explains essential programming concepts. And by the end of the book, kids have built simple games and created cool drawings with Python’s graphics library, Turtle. Each chapter closes with offbeat exercises that challenge the reader to put their newly acquired knowledge to the test.
The first thing that you need to know is this: If your computer has any sort of development environment set up on it, the instructions for installing Python provided in this book may be problematic or at least slightly difficult. I recommend using this book an an installation that is virtualized or simply a different computer than you otherwise develop on, not just so that your kid does not accidentally delete, or worse, alter and publish, your pet projects. Part of the process of modern programming, after all, is learning about the development environment.
There are a handful of good “learn to program in python” books out there and this one is similar; it is hard to know at which point someone using the book will pass from “Oh, I see, that’s easy” to “Huh?” which usually occurs a chapter or two after the person stopped paying attention to details. Python for Kids: A Playful Introduction to Programming does a good job of avoiding this problem by including a complete and rather extensive project, a game called the “Mr Sick Man Game” (which should be read “Mr. Stick-man game” and not “Mr. Stick… Mangame!”) There are plenty of other projects and individual programs that the book guides the reader through prior to the mangum stick opus. The book uses the “Turtle” module, based on LOGO, for much of this work. as well as the tkinter TH GUI toolkit interface. So if you don’t want your children near those modules, look for a different book, just in case you are involved in some sort of emacs-vim style code war.
Python for Kids is not available at this time but will be shipped in December, so this is a viable stocking stuffer option.
by Stefano Allesina and Madlen Wilmes, Princeton University Press.
But, if you use OpenSoucre software, you have both direct and indirect access to the actual technologies that are key to your science project. You can see exactly how the numbers are calculated or the graphic created, if you want to. It might not be easy, but at least you don’t have to worry about the first hurdle in looking under the hood that happens with commercial software: they won’t let you do it.
Another piece of advice. Some software decisions are based on deeply rooted cultural norms or fetishes that make no sense. I’m an emacs user. This is the most annoying, but also, most powerful, of all text editors. Here is an example of what is annoying about emac. In the late 70s, computer keyboards had a “meta” key (it was actually called that) which is now the alt key. Emacs made use of the metakey. No person has seen or used a metakey since about 1979, but emacs refuses to change its documentation to use the word “alt” for this key. Rather, the documentation says somethin like “here, use the meta key, which on some keyboards is the alt key.” That is a cultural fetish.
. But overall, this volume does a great job of mapping out the landscape of scripting-oriented scientific computing, using excellent examples from biology.
by Peter Farrell. Farrell is a math and computer science teacher who is interested in math education and using technology in learning. 
is an excellent example. 
is a new book-not-a-book that helps people who are staring out learning the Python programming language. 
. This is a book to help you learn middle level to more advanced Python, and at the same time, learn about codes, ciphers, and cryptography. The examples in the book tend to be classic examples, so this is a bit more like learning the history of encoding technologies and practices, and using Python as a way to play around with this interesting material. 
by super Python expert Al Sweigart is a pretty thick intermedia to somewhat advanced level programming book.
is a fast-paced, thorough introduction to programming with Python that will have you writing programs, solving problems, and making things that work in no time.
, but at close to half the price.
. The new Raspberry Pi 3 has features that make it much more useful and fun, including more speed, built in bluetooth, and built in wifi.
, or a related card, the other is the Raspberry Pi. If you look around you will find dozens of other circuit boards that do these things, but none of them have the huge support network, gazillion lines of pre-written code, and myriad bits of hardware designed to be added on to make things happen.
and a 
. I the Garage would be an Arduino with a movable camera and all that other stuff, controlled via the Pi that sits in my house.
, which came with the Pi 3 (with WiFi and Bluetooth, as mentioned), a power supply, a 32 GB Class 10 MicroSD card (you will want a class 10 card), a case, an HDMI cord, and two heat sinks. I think that normally lists for $90 but is perpetually on sale for much less.
comes with additional parts that will be helpful in building projects that exploit the Pi’s input/output pins, such as LED lights, hookup wires, and resistors, a breakout cable, and a breadboard. The way the Pi is set up, you will want a breakout system to allow you to mess with electronic components more easily and safely (safe for the Pi, that is, with its sensitive computer circuity). If I was going to give a kit to someone who I knew was going to mess around with the hardware, this is the kit I’d probably pick.
to hook up to that device, and you’d probably run it on your TV.
and 
.
. It comes in Lego form (
), and you can get a Minecraft cloud server version at 
. This book is intended to teach programming, in the Minecraft setting. The book is designed for kids 10 years and older, though I’m sure some younger kids can use it. Also, it must be admitted that a learning to program book like this may be most valuable for adults who are not coders but want to learn some coding, and happen to be gamers and like Minecraft.
by Jason Briggs comes in.
