2016 week 51 in programming
Python 3.6 released!
None
Bootstrap 4 drops IE9 support and goes full flexbox
None
The Programmer’s Guide to Booking a Concert
The simple keyword search would not allow me to write a query like “Return any user based in San Francisco or Oakland who has less than 10k followers and has posted at least one track”. I could write an algorithm which, when seeded with a Soundcloud user, would pull all their followers and followings, and then in turn pull all followers and followings for each of those users. Musicians regularly have tens or hundreds of thousands of followers. Pulling all followers for all users was obviously a sub-optimal approach. By keeping track of who follows who, I could later use an algorithm like PageRank to find up-and-comers who do not yet have many followers but nevertheless have a vote of confidence from the community. Analyzing the NetworkI wrote a Python script to query the database for any Soundcloud user with at least 500 followers, at least one track, and based in San Francisco or Oakland. We followed them through joyful and melancholic moments.
Creating an ELF Virus using Assembly
With even more luck, your virus would gain notoriety like the Whale Virus or the Michelangelo Virus. The way to achieve this is to: - change the entry point to the end of the text section - add the page size to the offset for the section header table - increase the file size and memory size of the text segment by the size of the virus code - for each program header that resides after the virus, increase the offset by the page size - find the last section header in the TEXT segment and increase the section size - for each section header that exists after the virus, increase the offset by the page size - insert the actual virus at the end of the text section - insert code that jumps to the original host entry point. The way we achieve this is: - add the virus size to the offset for the section header table, - in the text segment program header, decrease the virtual address by the size of the virus - in the text segment program header, increase the file size and memory size by the size of the virus - for each program header with an offset greater than the text segment, increase it by the size of the virus - change the entry point to the original text segment virtual address - the size of the virus - increase the program header offset by the size of the virus - insert the actual virus at the beginning of the text section. Increase the section header offset by the size of the virus - change the entry point to the end of the data segment - in the data segment program header, increase the page and memory size by the size of the virus - increase the bss offset by the size of the virus - set the executable permission bit on the DATA segment. A question you may ask yourself is, how does a virus grab its own code? How does a virus determine its own size? These are very good questions. Mov eax, 4 mov ecx, v start ; attach the virus portion mov edx, v stop - v start ; size of virus bytes int 80h. The size of the virus will calculate just fine, but the reference to the beginning of the virus code will fail after the first infection. Mov eax, 4 lea ecx, ; attach the virus portion mov edx, v stop - v start ; size of virus bytes int 80h. Notice that I didn’t include the system exit call in the virus.
The Death of CyanogenMod and What it Means for Development
Cyanogen Inc. has finally delivered a bullet to the brain - of CyanogenMod. CyanogenMod will no longer receive nightly builds after December 31st. But wait, what if the team simply finds another host to build nightlies? The CyanogenMod team will not continue official development on the project. Cyanogen Inc. owns the rights to the brand, so the CyanogenMod team has decided that it is no longer worth continuing development for the open source distribution without monetary or infrastructural support. Even if the team were to find an alternative revenue stream, the trouble isn’t worth it due to the potential legal issues that could be involved if Cyanogen Inc., and all brands that the company owns including CyanogenMod, were to be sold to another company. Plus, the team argues that the CyanogenMod brand has been tainted due to its association with Cyanogen, so new users may be wary of installing CyanogenMod. In this case, many devices may suffer a development drought as the development community largely relied on CyanogenMod to provide a stable branch that individuals could then fork. Whether or not the death of CyanogenMod spells the death of stable custom ROM development on certain devices will be up to developers and users alike.
JSON Schema Faker combines JSON Schema standard with fake data generators, allowing users to generate fake data that conform to the schema.
None
Linus Torvalds - What is acceptable for -ffast-math?
Most of the traditional heavy FP code tends to be much more about cache layout and good memory access patterns. Oh, round-to-zero is definitely acceptable in the world of “Who cares about IEEE, we want fast math, and we’ll use fixed arithmetic if the FP code is too slow”. What the hell do you call “Serious numerical coding”? Take a look at the computer game market today. It’s a lot more serious than most matematicians puttering around in their labs, let me tell you. The people you apparently consider serious are a lot more interested in fast communication and incredible memory bandwidth. I doubt you’ll find many of your “Serious numerical coding” people who would even notice the raw FP throughput. Look at SpecFP - CPU’s are fast enough, it spends most of its time waiting on memory.
A curated list of awesome bitwise operations and tricks
A curated list of awesome bitwise operations and tricks. These are techniques inspired by the fast inverse square root method. Frexp gives the 2n decomposition of a number, so that man, exp = frexp(x) means that man * 2exp = x and 0.5 <= man < 1. Fast nth Root of positive numbers via Infinite Series. Caveat: The 0x5c416 bias is given to center the method. If you plug in exp = -0.5, this gives the 0x5f3759df magic constant of the fast inverse root method. See these set of slides for a derivation of this method.
ThreadTone - half-tone circular loom image with threads
First a little image pre-processing is done to various input images for threading. To allow image processing on a pixel level we use openCV. Let’s get started and do some initial processing of the image. Import cv2 import numpy as np # Invert grayscale image def invertImage(image): return # Apply circular mask to image def maskImage(image, radius): y, x = np. Ogrid mask = x2 + y2 > radius**2 image[mask] = 0 return image # Load image image = cv2. Once the image is of the correct size it is converted to a gray-scale image and inverted. Int)-1). Using the functions above the algorithm iteratively adds lines to the image until either the maximum number of lines is reached or the stopping criteria is reached. Move over the image in question to see the original image.
The Art of Defensive Programming
A first sight to Defensive ProgrammingWhy do I think Defensive Programming is a good approach to issue these problems in certain kind of projects? There are many definitions for Defensive Programming, it also depends on the level of “Security” and level of resources you need for your software projects. Defensive programming practices are often used where high availability, safety or security is needed - Wikipedia. Let’s explore some of my diluted key points in order to achieve a Defensive Programming approach. Don’t trust developersDefensive programming can be related to something called Defensive Driving. In Defensive Driving we assume that everyone around us can potentially and possibly make mistakes. The same concept applies to Defensive Programming where us, as developers shouldn’t trust others developers’ code.
Modern garbage collection
Program throughput: how much does your algorithm slow the program down? This is sometimes expressed as a percentage of CPU time spent doing collection vs useful work. The first garbage collection algorithms were designed for uniprocessor machines and programs that had small heaps. In these cases you are probably willing to use an algorithm that actually slows down your program whilst it runs in order to do collection in the background and with low pause times. Warmup time: in response to the tuning issue, some collectors dynamically adapt the young generation size by observing how the program runs in practice, but now pause times depend on how long the program is running for as well. Put simply, the more memory your program uses the more slowly memory gets freed up, and the more time your computer spends doing collection vs useful work. Pause distribution: any garbage collector that runs concurrently with your program can encounter what the Java world calls a “Concurrent mode failure”: your program creates garbage faster than the GC threads can clean it up. Go optimises for pause times as the expense of throughput to such an extent that it seems willing to slow down your program by almost any amount in order to get even just slightly faster pauses.
Rocket: a web framework for Rust
None
An amazing set of resources for optimizing C++ and assembly for different processors and operating systems.
Optimizing software in C++: An optimization guide for Windows, Linux and Mac platforms This is an optimization manual for advanced C++ programmers. Optimizing subroutines in assembly language: An optimization guide for x86 platforms This is an optimization manual for advanced assembly language programmers and compiler makers. The microarchitecture of Intel, AMD and VIA CPUs: An optimization guide for assembly programmers and compiler makers This manual contains details about the internal working of various microprocessors from Intel, AMD and VIA. Topics include: Out-of-order execution, register renaming, pipeline structure, execution unit organization and branch prediction algorithms for each type of microprocessor. Intended as a source of reference for programmers who want to make function libraries compatible with multiple compilers or operating systems and for makers of compilers and other development tools who want their tools to be compatible with existing tools. Supports many different compilers under Windows, Linux, BSD and Mac OS X operating systems, 32 and 64 bits. The ForwardCom instruction set has variable-length vector registers and a special addressing mode that allows the software to automatically adapt to different microprocessors with different maximum vector lengths and make efficient loops through arrays regardless of whether the array size is divisible by the vector length. Pdf, size: 461263, last modified: 2016-Dec-08.Download. Test programs that I have used for my research.
BearSSL - Constant-Time Crypto
Constant-time implementations are pieces of code that do not leak secret information through timing analysis. Even if a strong RNG is available at some level, bringing it to the implementation of a nominally deterministic algorithm can be troublesome in terms of internal API. For these reasons, BearSSL aims for constant-time implementations for all algorithms, or at least all implementations used by default. Using a signed type would be a bit weird, but it would make some sort of sense if the developer is in fact converting an existing implementation in Java to C code, because Java does not have unsigned types, and guarantees modular arithmetics. A very good source on how to implement constant-time operations in C is the Cryptography Coding Standard, a collaborative effort for describing issues related to secure implementation of cryptographic algorithms, and, more generally, of code that processes secret data elements. The “Big” implementation is a classic, table-based implementation, which is not constant-time. The “Size” values correspond to the total code footprint for CBC and CTR implementations, respectively. Current BearSSL implementation of elliptic curve cryptography uses the same generic big integer code as RSA. It thus inherits its constant-time characteristics.
autovpn - a Script to create an OpenVPN Endpoint on AWS
Script that allows the easy creation of OpenVPN endpoints in any AWS region. It spins up a tagged ec2 instance and configures OpenVPN software. Once instance is configured an OpenVPN configuration file is downloaded and ready to use. Additional functionality includes specifying instance type, generate ssh keypairs, specify custom ami, change login user and more to come. Execute autovpn with -C -k and -r options to deploy to AWS./autovpn -C -r us-east-1 -k macbook OpenVPN config files are downloaded to current working directory. EXAMPLES: Create OpenVPN endpoint: autovpn -C -r us-east-1 -k macbook Generate keypair in a region. Autovpn -G -r us-east-1 Get running instances autovpn -S -r us-east-1 Terminate OpenVPN endpoint autovpn -T -r us-east-1 -z i-b933e00c Using custom options autovpn -C -r us-east-1 -k macbook -a ami-fce3c696 -u ec2 user -i m3.
Adopt Python 3
In the midst of all the celebration, many of you were still asking if it is safe to drop Python 2 and move over to Python 3. Let’s start with popular packages i.e packages that have the highest download counts on PyPI. Python 3 Wall of Superpowers and Python 3 Readiness are two websites that maintain a list of 200 and 360 most popular packages respectively. If you open these sites today, you will see that 187/200, and 341/360 packages support Python 3. Total Python 3 coverage is at 72 %. That’s not so bad given that Python 3 came out in 2008 and 2020 is the official EOL of Python 2.7. The small packages shouldn’t hold anyone back from Python 3, because if you desperately need them at some point, you should be able to port them yourself without much overhead. A whopping 75 % of such Python2-only packages are small and easy-to-port. If you need them and there isn’t a Python 3 compatible alternative, then you are stuck with Python 2. You can adopt Python 3 and enjoy all the goodies that come along with it!
Hacksaw Academy - Interactive platform where you can quickly learn HTML, CSS, and Javascript by building functional apps
None
Announcing Rust 1.14
The Rust team is happy to announce the latest version of Rust, 1.14.0. Rust is a systems programming language focused on safety, speed, and concurrency. As always, you can install Rust 1.14.0 from the appropriate page on our website, and check out the detailed release notes for 1.14.0 on GitHub. One of the biggest features in Rust 1.14 isn’t actually in the language or compiler: the rustup tool has reached a 1.0 release, and is now the recommended way to install Rust from the project directly. Rustup installs The Rust Programming Language from the official release channels, enabling you to easily switch between stable, beta, and nightly compilers and keep them updated. The community has been doing interesting, experimental work in this area: see Jan-Erik’s slides for the workshop he ran at Rust Belt Rust for some examples, or check out Tim’s example of the classic TodoMVC project. Xargo allows for easy cross-compilation of Rust to bare-metal targets.
Google employee sues company for ‘illegal’ confidentiality policies that violate labor laws
A Google product manager has accused the company of violating California labor laws via its restrictive confidentiality policies. According to The Information, the employee filed a suit with the California Superior Court in San Francisco, alleging that Google is running an internal “Spying program” that encourages employees to report co-workers suspected of leaking information to the media. The lawsuit also states that Google’s policies prohibit employees from reporting illegal activities within the company, even to its own attorneys. Weirdly, there’s also a policy that prevents employees from writing a novel about working for a Silicon Valley corporation without getting approval from Google. One of the reasons for the stringent policies is to ensure that confidential information isn’t leaked to the press. The suit also says that confidential information is classified as “Everything at Google,” which prevents employees from talking about their workplace conditions with the “Press, members of the investment community, partners, or anyone else outside of Google.” If Google is found to be guilty of the alleged 12 violations of California’s labor laws, it could pay out as much as $3.8 billion in total, with 75% of the penalty collected by the state and the rest distributed to Google’s 65,000 employees.
We’re bringing GitLab Pages to the Community Edition
The wider community asked us open source GitLab pages. Inspired by the holiday-spirit we’re happy to bring GitLab Pages to the Community Edition of GitLab. GitLab Pages allows you to host static websites straight from GitLab, with any kind of static site generator. My personal website is hosted through GitLab Pages on GitLab.com using my favorite static site generator Middleman. Until today, GitLab Pages was exclusive to the Enterprise Edition of GitLab. More than a hundred people voted and discussed bringing GitLab Pages to our open-source MIT-licensed Community Edition. In this case, I’m happy to announce that we will be bringing GitLab Pages to the Community Edition.
Cryptography Coding Standard
Welcome to the Cryptography Coding Standard homepage. The Cryptography Coding Standard is a set of coding rules to prevent the most common weaknesses in software cryptographic implementations. CCS was first presented and discussed at the Internet crypto workshop on Jan 23, 2013. Coding rules: the list of coding rules, with for each rule a statement of the problem addressed and one or more proposed solutions. These pages can also be accessed with the navigation bar on the left.
Computer Science from the Bottom Up
None
8bitworkshop: Online 6502 IDE that targets the Atari 2600
Right now it supports the Atari 2600/VCS. Enter 6502 source code on the left, and we’ll assemble it and show you the result on the right. You can also select a sample program from the pulldown at the top-left of the page.
Is mathematics the oldest legacy system?
The design of mathematics encompasses a number of other principles that are also present in software engineering. Let’s now take a look at the real number system. Throw in the operations of multiplication and division, and we get the rational numbers, which is now a field. The story might end here, and we’d all be happy: we have a bunch of numbers and a bunch of operations, we can apply those operations to those numbers and still end up with the same set of numbers. In the end, one might say that there is nothing real about the real numbers - it’s all a construction! From an alternative perspective, mathematics contains striking parallels with software engineering. By becoming more familiar with the landscape of mathematics, we can help with the cross pollination of ideas between mathematics and software engineering.
Is this still relevant? “GCE vs AWS in 2016: Why you should NEVER use Amazon!”
We are running hundreds of instances on AWS, and we’ve been doing so for some time, growing at a sustained pace. The following highlights many issues encountered day to day on AWS so that [hopefully] you don’t do the same mistakes we’ve done by picking AWS. There are a lot of clouds: GCE, AWS, Azure, Digital Ocean, RackSpace, SoftLayer, OVH, GoDaddy Check out our article Choosing a Cloud Provider: AWS vs GCE vs SoftLayer vs DigitalOcean vs. We’ll focus only on GCE and AWS in this article. Base instance plus storage cost Add provisioned IOPS for databases Add local SSD Add 10% on top of everything for Premium Support Add 10% for dedicated instances or dedicated hosts. Base instance plus storage cost Enjoy fast and dependable IOPS out-of-the-box on remote SSD volumes Add local SSD Enjoy automatic discount for sustained usage AWS IO are expensive and inconsistent EBS SSD volumes: IOPS, and P-IOPS. We are forced to pay for Provisioned-IOPS whenever we need dependable IO. The P-IOPS are NOT really faster. Local SSD storage is only available via the i2 instances family which are the most expensive instances on AWS. There is no granularity possible. “Dedicated instances are Amazon EC2 instances that run in a virtual private cloud on hardware that’s dedicated to a single customer. Your Dedicated instances are physically isolated at the host hardware level from your instances that aren’t Dedicated instances and from instances that belong to other AWS accounts.” What GCE does by comparison is a PURELY AWESOME MONTHLY AUTOMATIC DISCOUNT. Instances hours are counted at the end of every month and discount is applied automatically.