It's not, strictly speaking, the furthest back in time that we've ever seen (at least, I don't think so -- my understanding was that the Cosmic Microwave Background radiation holds that record). That pedantic point aside, though, it's still a remarkable achievement and hopefully the next generation of observatories will continue to push back the limits of astronomy. I think seeing this galaxy is closer to discovering your first dinosaur fossil if all you'd previously seen of life on earth was sea life in the ancient Cambrian oceans -- ie no less impressive, arguably more so.
We have an odd understanding of the history of our Universe. Starting from the Big Bang, we can only speculate about what came "before" it, and in the first tiny fraction of a second or so, then we (think we) know what went on in the next few seconds, then for a couple of hundred thousand years it's all blank again (at least experimentally), then again a weird spike of information, and then another gap, and so on... so much room to fill for understanding. The state of the Universe after CMB but before "normal" galaxy formation is one of the larger gaps and one that this goes some way towards filling.