Category Archives: Algorithm

Coders

Source: Marginal Revolution, Mar 2019

The list of small-person or one-person innovators is long…[long list follows]…

The reason so few people can have such an outsize impact, Andreessen argues, is that when you’re creating a weird new prototype of an app, the mental castle building is most efficiently done inside one or two isolated brains. 

The 10X productivity comes from being in the zone and staying there and from having a remarkable ability to visualize a complex architecture. 

“If they’re physical capable of staying awake, they can get really far,” he says.  “The limits are awake time.  It takes you two hours to get the whole thing loaded into your head, and then you get like 10 or 12 or 14 hours where you can function at that level.”  The 10Xers he has known also tend to be “systems thinkers,” insatiably curious about every part of the technology stack, from the way currents flow in computer processors to the latency of touchscreen button presses.  “It’s some combination of curiosity, drive, and the need to understand.  They find it intolerable if they don’t understand some part of how the system works.”

Machine learning ‘causing science crisis’

Source: BBC, Feb 2019

A growing amount of scientific research involves using machine learning software to analyse data that has already been collected. This happens across many subject areas ranging from biomedical research to astronomy. The data sets are very large and expensive.

‘Reproducibility crisis’

But, according to Dr Allen, the answers they come up with are likely to be inaccurate or wrong because the software is identifying patterns that exist only in that data set and not the real world.

“Often these studies are not found out to be inaccurate until there’s another real big dataset that someone applies these techniques to and says ‘oh my goodness, the results of these two studies don’t overlap‘,” she said.

“There is general recognition of a reproducibility crisis in science right now. I would venture to argue that a huge part of that does come from the use of machine learning techniques in science.”

The “reproducibility crisis” in science refers to the alarming number of research results that are not repeated when another group of scientists tries the same experiment. It means that the initial results were wrong. One analysis suggested that up to 85% of all biomedical research carried out in the world is wasted effort.

Flawed patterns

Machine learning systems and the use of big data sets has accelerated the crisis, according to Dr Allen. That is because machine learning algorithms have been developed specifically to find interesting things in datasets and so when they search through huge amounts of data they will inevitably find a pattern.

“The challenge is can we really trust those findings?” she told BBC News.

“Are those really true discoveries that really represent science? Are they reproducible? If we had an additional dataset would we see the same scientific discovery or principle on the same dataset? And unfortunately the answer is often probably not.”

Data > AI Framework/Architecture

Source: Medium, Feb 2019

At its essence, machine learning works by extracting information from a dataset and transferring it to the model weights. A better model is more more efficient at this process (in terms of time and/or overall quality), but assuming some baseline of adequacy (that is, the model is actually learning something) better data will trump a better architecture.

To illustrate this point, let’s have a quick and dirty test. I created two simple convolutional networks, a “better” one, and a “worse” one. The final dense layer of the better model had 128 neurons, while the worse one had to make due with only 64. I trained them on subsets of the MNIST dataset of increasing size, and plotted the models’ accuracy on the test set vs the number of samples they were trained on.

The positive effect of training dataset size is obvious (at least until the models start to overfit and accuracy plateaus). My “better” model, blue line, clearly outperforms the “worse” model, green line. However, what I want to point out is that the accuracy of the “worse” model trained on 40 thousand samples is better than of the “better” model at 30 thousand samples!

Good engineering is always important, but if you are doing AI, the data is what creates the competitive advantage. The billion dollar question is, however, if you are going to be able to maintain your advantage.

Creating a dataset, however, is a different sort of problem. Usually, it requires a lot of manual human labor — and you can easily scale it by hiring more people. Or it could be that someone has the data — then all you have to do is pay for a license. In any case — the money makes it go a lot faster.

A much better option is to treat AI as a lever. You can take an existing, working business model and supercharge it with AI. For example, if you have a process which depends on human cognitive labor, automating it away will do wonders for your gross margins. Some examples I can think of are ECG analysisindustrial inspectionsatellite image analysis. What is also exciting here is that, because AI stays in the backend, you have some non-AI options to build and maintain your competitive advantage.

Building AI models may be very interesting, but what really matters is having better data than the competition. Maintaining a competitive advantage is hard, especially if you encounter a competitor who is richer than you, which is very likely to happen if your AI idea takes off. You should aim to create a scalable data collection process which is hard to reproduce by your competition. AI is well suited to disrupt industries which rely on the cognitive work of low qualified humans, as it allows to automate this work.

Godel & Turing

Source: British Computer Society, Jun 2004/Nov 2007

His practical proposals for digital computing were streets ahead in their recognition of non-numerical structures.

Turing’s introduction stated that he hoped soon to base ‘the theory of
functions of a real variable’ on this concept, and this remark suggests that in 1936 he had every intention of redefining the relationship between Z and R in a constructivist way.

it is due to Turing’s quite inexplicable genius that he saw a connection, invisible to others, between Hilbert’s decision problem, the classical question of free will and physical determinism, and the computation of real numbers.3

G¨odel had shown how to encode as integers, statements about integers, and thereby showed the incompleteness of formal systems.

Turing showed how to encode operations on integers, by integers, and thereby showed the undecidability of formal systems. Thus Turing had the vital perception that operations and numerical data could be codified alike as a symbolic structure.

in a remarkable application of that perception, Turing showed that ‘it is possible to invent a single machine which can be used to compute any computable sequence’. This invention was his universal machine.

Turing’s mechanization of judgment through ‘weight of evidence’ (anticipating statistical methods very popular in Artificial Intelligence sixty years later) was perhaps a key step in persuading him of the boundless scope for the mechanization of human intelligence. In 1943, he discussed such ideas with Shannon, and in 1944 he was speaking of ‘building a brain’.

But Turing’s post-1945 views still marked a definite shift. In particular they implied a rejection of the view that uncomputable ‘intuitive’ steps were needed for the recognition of true but formally unprovable G¨odel statements.

Such uncomputable mental steps were implied by his 1938 analysis, and G¨odel himself certainly believed in them. One sentence in Turing’s proposal for the ACE computer (Turing 1946) is evidence of this shift: it is a claim that a computer could play ‘very good chess’ if ‘occasional serious mistakes’ were allowed. This cryptic reference to mistakes was clarified by the argument of (Turing 1947) that once the possibility of mistakes is allowed, G¨odel’s theorem becomes irrelevant.

(G¨odel, of course, rejected this post-war Turing vision of machine intelligence, although he had endorsed Turing’s original definition of computability.)

That a program is itself a form of data is the basis of modern computing. We can
now hardly imagine a world in which this idea was not familiar, and it is hard to read On computable numbers without translating Turing machines into programs and the universal machine into a computer running those programs. Turing’s 1936 construction of the universal machine, by treating programs as data to be read in just the same way as numerical or any other kind of symbolic data, defined a structure quite different from that of Babbage and his successors up to the ENIAC.

Turing certainly saw from the very start that programs were data and could be manipulated as such. On the first page of his 1946 report, Turing explained that the management of a calculation would be ‘looked after by the machine itself’

Turing also wrote in a most important passage that the future of software writing should be free from drudgery because ‘any processes that are quite mechanical may be turned over to the machine computer itself.’ This insight (going far beyond von Neumann) depended on his perception of programs being themselves data-structures. In (Turing 1947) he spoke of how a machine could be given instructions in ‘any symbolic logic’.

The thrust of these post-war papers, putting forward what he called a ‘heretical theory’ of the mind, was to put forward a wholly computable model. Penrose, summarizing Turing’s position in 1950, says that ‘It seems likely that he viewed physical action in general—which would include the action of a human brain—to be always reducible to some kind of Turing-machine action’ (Penrose 1994, p. 21). This goes a little further than what Turing explicitly wrote in 1950, but reflects what is implicit in the entire framework.

A late-1950s Turing might also, in the light of his work in morphogenesis, seeing discrete structures emerging out of non-linear equations, have taken more interest in exploring the relationship of computability with continuous dynamical systems

when computing is front-page news, physics has lost its 1950s status. But
computing requires both software and hardware, and the Church-Turing thesis, which underlies computing, relates both to logic and to physics

… from 1945 onwards he took the view that all mental processes fell within the scope of computable operations.

Google and FB Depend upon Advertising

Source: Visual Capitalist, May 2017

Related Resource: Visual Capitalist, Sep 2017

GE’s Digital Twins

Source: MIT Technology Review, Jul 2017

cloud-hosted software models of GE’s machines that can be used to save money and improve safety for its customers. GE builds these “digital twins”using information it gathers from sensors on the machines, supplemented with physics-based models, AI, data analytics, and knowledge from its scientists and engineers. Though digital twins are primarily lines of software code, the most elaborate versions look like 3-D computer-aided design drawings full of interactive charts, diagrams, and data points. They enable GE to track wear and tear on its aircraft engines, locomotives, gas turbines, and wind turbines using sensor data instead of assumptions or estimates, making it easier to predict when they will need maintenance. An aircraft engine flying over the U.S. could, for instance, have a digital twin on a GE computer server in California help determine the best service schedule for its parts.

The technology depends on artificial intelligence to continually update itself. What’s more, if data is corrupted or missing, the company fills in the gaps with the aid of machine learning, a type of AI that lets computers learn without being explicitly programmed, says Colin Parris, GE Global Research’s vice president for software research. Parris says GE pairs computer vision with deep learning, a type of AI particularly adept at recognizing patterns, and reinforcement learning, another recent advance in AI that enables machines to optimize operations, to enable cameras to find minute cracks on metal turbine blades even when they are dirty and dusty.

Collective Computation @ Santa Fe Institute

Source: Quanta Magazine, Jul 2017

Flack’s focus is on information: specifically, on how groups of different, error-prone actors variously succeed and fail at processing information together. “When I look at biological systems, what I see is that they are collective,” she said. “They are all made up of interacting components with only partly overlapping interests, who are noisy information processors dealing with noisy signals.”

How did you get into research on problem solving in nature, and how did you wind up at the Santa Fe Institute?

I’ve always been interested in how nature solves problems and where patterns come from, and why everything seems so organized despite so many potential conflicts of interest.

Collective computation is about how adaptive systems solve problems. All systems are about extracting energy and doing work, and physical systems in particular are about that. When you move to adaptive systems, you’ve got the additional influence of information processing, which we think allows a system to extract energy more efficiently even though it has to expend a little extra energy to do the information processing. Components of adaptive systems look out at the world, and they try to discover the regularities. It’s a noisy process.

Unlike in computer science where you have a program you have written, which has to produce a desired output, in adaptive systems this is a process that is being refined over evolutionary or learning time. The system produces an output, and it might be a good output for the environment or it might not. And then over time it hopefully gets better and better.

We have this principle of collective computation that seems to involve these two phases. The neurons go out and semi-independently collect information about the noisy input, and that’s like neural crowdsourcing. Then they come together and come to some consensus about what the decision should be. And this principle of information accumulation and consensus applies to some monkey societies also. The monkeys figure out sort of semi-independently who is capable of winning fights, and then they consolidate this information by exchanging special signals. The network of these signals then encodes how much consensus there is in the group about any one individual’s capacity to use force in fights.

Now that you can follow up on these kinds of questions to your heart’s content, what would you say if you could visit yourself back at Cornell, in the stacks of the library?

Jorge Luis Borges is one of my favorite writers, and he wrote something along the lines of “the worst labyrinth is not that intricate form that can trap us forever, but a single and precise straight line.” My path is not a straight line. It has been a quite interesting, labyrinthine path, and I guess I would say not to be afraid of that. You don’t know what you’re going to need, what tools or concepts you’re going to need. The thing is to read broadly and always keep learning.

Can you talk a bit about what it’s like to start with a table of raw data and pull these sorts of grand patterns out of it? Is there a single eureka moment, or just a slow realization?

Typically what happens is, we have some ideas, and our group discusses them, and then over months or years in our group meetings we sort of hash out these issues. We are ok with slow, thoughtful science. We tend to work on problems that are a little bit on the edge of science, and what we are doing is formalizing them. A lot of the discussion is: “What is the core problem, how do we simplify, what are the right measurements, what are the right variables, what is the right way to represent this problem mathematically?” It’s always a combination of the data, these discussions, and the math on the board that leads us to a representation of the problem that gives us traction.

I believe that science sits at the intersection of these three things — the data, the discussions and the math. It is that triangulation — that’s what science is. And true understanding, if there is such a thing, comes only when we can do the translation between these three ways of representing the world.